Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Question about running Hive on Tez


Copy link to this message
-
Question about running Hive on Tez
Hi,

Excuse me. May I ask a question about running Hive on Tez?

I've installed Hive on Tez, and running a simple query from hiveCli,

hive> set hive.optimize.tez=true;
hive> select * from table order by title_id limit 5;

While, each time, I could see from the TezJobMonitor that, all the map
jobs are done, but the reducer never get started, and the job is
running forever there.

I tried a number of times, and each time the same failure(job running
hangs) happens again and again.
Does anyone successfully running queries using Hive on Tez? Are there
any tips or suggestions?

Here is my job log:

13/12/13 20:57:31 INFO client.TezSession: Submitting dag to
TezSession, sessionName=HIVE-365b35bc-2461-4e2f-83f9-8da1fa356a86,
applicationId=application_1386896881353_0027
13/12/13 20:57:33 INFO client.TezSession: Submitted dag to TezSession,
sessionName=HIVE-365b35bc-2461-4e2f-83f9-8da1fa356a86,
applicationId=application_1386896881353_0027,
dagId=dag_1386896881353_0027_1
13/12/13 20:57:33 INFO client.RMProxy: Connecting to ResourceManager
at /10.183.195.180:9022
13/12/13 20:57:33 INFO log.PerfLogger: </PERFLOG method=TezSubmitDag
start=1386968251250 end=1386968253338 duration=2088
from=org.apache.hadoop.hive.ql.exec.tez.TezTask>
13/12/13 20:57:33 INFO tez.TezJobMonitor:

13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG method=TezRunDag
from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG
method=TezSubmitToRunningDag
from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
13/12/13 20:57:33 INFO log.PerfLogger: </PERFLOG
method=TezSubmitToRunningDag start=1386968253341 end=1386968253402
duration=61 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
Status: Running (application id: application_1386896881353_0027)

13/12/13 20:57:33 INFO tez.TezJobMonitor: Status: Running (application
id: application_1386896881353_0027)

13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG
method=TezRunVertex.Reducer 2
from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
13/12/13 20:57:33 INFO log.PerfLogger: <PERFLOG
method=TezRunVertex.Map 1
from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
Map 1: -/- Reducer 2: -/-
13/12/13 20:57:33 INFO tez.TezJobMonitor: Map 1: -/- Reducer 2: -/-
Map 1: -/- Reducer 2: 0/1
13/12/13 20:57:33 INFO tez.TezJobMonitor: Map 1: -/- Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:34 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:37 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:40 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:43 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:46 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:49 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:52 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 0/16 Reducer 2: 0/1
13/12/13 20:57:55 INFO tez.TezJobMonitor: Map 1: 0/16 Reducer 2: 0/1
Map 1: 1/16 Reducer 2: 0/1
13/12/13 20:57:56 INFO tez.TezJobMonitor: Map 1: 1/16 Reducer 2: 0/1
Map 1: 2/16 Reducer 2: 0/1
13/12/13 20:57:58 INFO tez.TezJobMonitor: Map 1: 2/16 Reducer 2: 0/1
Map 1: 3/16 Reducer 2: 0/1
13/12/13 20:57:58 INFO tez.TezJobMonitor: Map 1: 3/16 Reducer 2: 0/1
Map 1: 5/16 Reducer 2: 0/1
13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 5/16 Reducer 2: 0/1
Map 1: 8/16 Reducer 2: 0/1
13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 8/16 Reducer 2: 0/1
Map 1: 12/16 Reducer 2: 0/1
13/12/13 20:57:59 INFO tez.TezJobMonitor: Map 1: 12/16 Reducer 2: 0/1
Map 1: 15/16 Reducer 2: 0/1
13/12/13 20:58:00 INFO tez.TezJobMonitor: Map 1: 15/16 Reducer 2: 0/1
13/12/13 20:58:00 INFO log.PerfLogger: </PERFLOG
method=TezRunVertex.Map 1 start=1386968253402 end=1386968280223
duration=26821 from=org.apache.hadoop.hive.ql.exec.tez.TezJobMonitor>
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:00 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:03 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:06 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:09 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:12 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:15 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:18 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:21 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:24 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:27 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:30 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:33 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:36 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:39 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:42 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:45 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:48 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:51 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:54 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:58:57 INFO tez.TezJobMonitor: Map 1: 16/16 Reducer 2: 0/1
Map 1: 16/16 Reducer 2: 0/1
13/12/13 20:59:00 INFO tez.TezJobMonitor:
+
Gunther Hagleitner 2013-12-14, 00:51
+
Cheolsoo Park 2013-12-16, 18:25
+
Lefty Leverenz 2013-12-19, 00:38