Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Hive 0.7.1 Query hands


Hello!

I use Hive 0.7.1 over Hadoop 0.20.2 (CHD3u3) on 70 nodes cluster.
I have a trouble with query like this:

*FROM* (  *SELECT* *id*, {expressions} *FROM* table1   *WHERE*
day='2013-02-16' *AND* ({conditions1})*UNION* *ALL*  *SELECT* *id*,
{expressions} *FROM* table2   *WHERE* day='2013-02-16' *AND*
(conditions)*UNION* *ALL*  *SELECT* *id*, {expressions} *FROM* table3
*WHERE* day='2013-02-16' *AND* (conditions)*UNION* *ALL*  *SELECT*
*id*, {expressions} *FROM* table4  *WHERE* day='2013-02-16' *AND*
(conditions)) union_tmp*INSERT* OVERWRITE *table* result_table
*PARTITION* (day='2013-02-16')*SELECT* *id*, transformations
(expressions)*GROUP* *BY* *id*;

it had 4865 map tasks and 100 reduce tasks.
first 4780 map taks completed succefull and last
85 tasks hangs.

All this tasks hands with no progress,
and no tasks attempts for each:

One hour later after hang situation starting, job failed with exeption:

2013-02-20 20:02:02,000 Stage-1 map = 0%,  reduce = 0%
2013-02-20 20:02:40,679 Stage-1 map = 1%,  reduce = 0%
2013-02-20 20:02:54,022 Stage-1 map = 2%,  reduce = 0%
2013-02-20 20:03:14,129 Stage-1 map = 3%,  reduce = 0%
......................................................
2013-02-20 21:18:00,361 Stage-1 map = 98%,  reduce = 22%
2013-02-20 21:18:05,691 Stage-1 map = 98%,  reduce = 23%
java.io.IOException: Call to statlabjt/10.6.0.55:8021 failed on local
exception: java.io.EOFException
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
at org.apache.hadoop.ipc.Client.call(Client.java:1110)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
at org.apache.hadoop.mapred.$Proxy8.getJobStatus(Unknown Source)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1053)
at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1065)
at org.apache.hadoop.hive.ql.exec.ExecDriver.progress(ExecDriver.java:351)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:672)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:425)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
Caused by: java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:375)
at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:815)
at org.apache.hadoop.ipc.Client$Connection.run(Client.java:724)
Ended Job = job_201302152355_4764 with exception
'java.io.IOException(Call to statlabjt/10.6.0.55:8021 failed on local
exception: java.io.EOFException)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.MapRedTask

How to find reasons of such situation?
How to prevent such situations in future?

Best regards
Mezentsev Pavel
+
Jarek Jarcec Cecho 2013-02-21, 15:59
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB