Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive NR map progress inconsistent and regurlarly restart from 0%


Copy link to this message
-
Hive NR map progress inconsistent and regurlarly restart from 0%
I have a Yarn MR (with two ec2 instances to mapreduce) job on a dataset of approximately a thousand avro records, and the map phase is behaving erratically. See the progress below

What is going on there ?

            hive> select * from nikon where qs_cs_s_aid='VIEW' limit 10;

            Total MapReduce jobs = 1
            Launching Job 1 out of 1
            Number of reduce tasks is set to 0 since there's no reduce operator
            Starting Job = job_1352281315350_0020, Tracking URL = http://blabla.ec2.internal:8088/proxy/application_1352281315350_0020/
            Kill Command = /usr/lib/hadoop/bin/hadoop job  -Dmapred.job.tracker=blabla.com:8032 -kill job_1352281315350_0020
            Hadoop job information for Stage-1: number of mappers: 4; number of reducers: 0

            2012-11-07 11:14:40,976 Stage-1 map = 0%,  reduce = 0%
            2012-11-07 11:15:06,136 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 10.38 sec
            2012-11-07 11:15:07,253 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 12.18 sec
            2012-11-07 11:15:08,371 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 12.18 sec
            2012-11-07 11:15:09,491 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 12.18 sec
            2012-11-07 11:15:10,643 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 15.42 sec
            (...)
            2012-11-07 11:15:35,441 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 37.77 sec
            2012-11-07 11:15:36,486 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 37.77 sec

here restart at 16% ?

            2012-11-07 11:15:37,692 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 21.15 sec
            2012-11-07 11:15:38,815 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 21.15 sec
            2012-11-07 11:15:39,865 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 21.15 sec
            2012-11-07 11:15:41,064 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 22.4 sec
            2012-11-07 11:15:42,181 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 22.4 sec
            2012-11-07 11:15:43,299 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 22.4 sec

here restart at 0% ?

            2012-11-07 11:15:44,418 Stage-1 map = 0%,  reduce = 0%
            2012-11-07 11:16:02,076 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 6.86 sec
            2012-11-07 11:16:03,193 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 6.86 sec
            2012-11-07 11:16:04,259 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 8.45 sec
            (...)
            2012-11-07 11:16:31,291 Stage-1 map = 22%,  reduce = 0%, Cumulative CPU 35.34 sec
            2012-11-07 11:16:32,414 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 37.93 sec

here restart at 11% ?

            2012-11-07 11:16:33,459 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU 19.53 sec
            2012-11-07 11:16:34,507 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU 19.53 sec
            2012-11-07 11:16:35,731 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 21.47 sec
            (...)
            2012-11-07 11:16:46,839 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU 24.14 sec

here restart at 0% ?

            2012-11-07 11:16:47,939 Stage-1 map = 0%,  reduce = 0%
            2012-11-07 11:16:56,653 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 7.54 sec
            2012-11-07 11:16:57,814 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 7.54 sec
            (...)

Needless to say the job crashes after some time with an Error: java.io.IOException: java.io.IOException: java.lang.ArrayIndexOutOfBoundsException: -56

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB