Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException


Copy link to this message
-
Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException
I'm having exactly this problem, and it's causing my job to fail when I try
to process a larger amount of data (I'm attempting to process 30GB of
compressed CSVs and the entire job fails every time).

This issues is open for it:
https://issues.apache.org/jira/browse/MAPREDUCE-5

Anyone have any idea about a workaround for the problem? To my eyes Hadoop
is just crashing when I try to process a large job (v1.0.3 on Elastic
MapReduce). But this just seems crazy, there must be something I can do to
get things working.

The only difference between what is stated in that bug report and mine is
that some of my map tasks fail at the end, but I believe that is due to the
reduce tasks causing problems because the map tasks are just timing out
without much more information than that.

Description (copied from JIRA):
---------------------------------------
During the shuffle phase, I'm seeing a large sequence of the following
actions:
1) WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_200905181452_0002_m_000010_0,0) failed :
org.mortbay.jetty.EofException
2) WARN org.mortbay.log: Committed before 410
getMapOutput(attempt_200905181452_0002_m_000010_0,0) failed :
org.mortbay.jetty.EofException
3) ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException:
Committed
The map phase completes with 100%, and then the reduce phase crawls along
with the above errors in each of the TaskTracker logs. None of the
tasktrackers get lost. When I run non-data jobs like the 'pi' test from the
example jar, everything works fine.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB