Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Hive job got stuck in the last task


Copy link to this message
-
Hive job got stuck in the last task
MiaoMiao 2012-08-10, 06:29
Hi there,
I've been doing a few pre-research of hive for my project, and met a
problem recently.

I tried this query in hive cli, "select count(1) from
day_20120629_main a join day_20120331_main b on
(a.eadc_type=b.eadc_type);",
but the hive job has been stuck in the very last task for 69hrs and
still unfinished. If my project moves to hive, and the queries stuck
every now and then, it will be unacceptable, so I want to know what is
going on and how to solve the case.

The whole job was divided into 757 map tasks and 51 reduce tasks, most
maps finished in less than 1 minute. While other reduces' durations
varies from 1 minute to 6 hours, and the last reduce has been running
for 67 hours.

Below is some info that may help debug.

day_20120629_main and day_20120331_main are both external tables, each
of which points to a 20GB file.
There are 700 fields in each table, with various types. Here I listed
some of them.
FIELD TYPE
ID bigint
eadc_type string
eadc_id string
Counters for task_201207301017_0173_r_000001 (Finished in 5hrs, 51mins, 57sec)
org.apache.hadoop.hive.ql.exec.Operator$ProgressCounter
CREATED_FILES 1

FileSystemCounters
FILE_BYTES_READ 1,654,915
FILE_BYTES_WRITTEN 1,749,446
HDFS_BYTES_WRITTEN 119

org.apache.hadoop.hive.ql.exec.JoinOperator$SkewkeyTableCounter
SKEWJOINFOLLOWUPJOBS 0

Map-Reduce Framework
Reduce input groups 1,628,710
Combine output records 0
Reduce shuffle bytes 2,495,327
Reduce output records 0
Spilled Records 1,628,710
Combine input records 0
Reduce input records 1,628,710

Counters for task_201207301017_0173_r_000000 (The one lasts for 67 hrs
and still running)

FileSystemCounters
FILE_BYTES_READ 1,966,080
FILE_BYTES_WRITTEN 2,657,477

Map-Reduce Framework
Reduce input groups 3,078,314
Combine output records 0
Reduce shuffle bytes 3,212,839
Reduce output records 0
Spilled Records 0
Combine input records 0
Reduce input records 3,078,314