Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Mapper always hangs at the same spot


Copy link to this message
-
Re: Mapper always hangs at the same spot
When you say "never progresses", do you see the MR framework kill it
automatically after 10 minutes of inactivity or does it never ever
exit?

You can lower the timeout period on tasks via mapred.task.timeout set
in msec. You could also set mapred.max.map.failures.percent to a
non-zero value to allow that much percentage of tasks to fail without
also marking the whole job as a failure.

If the task itself does not get killed by the framework due to
inactiveness, try doing a hadoop job -fail-task on its attempt ID
manually.

On Sat, Apr 13, 2013 at 8:45 PM, Chris Hokamp <[EMAIL PROTECTED]> wrote:
> Hello,
>
> We have a job where all mappers finish except for one, which always hangs at
> the same spot (i.e. reaches 49%, then never progresses).
>
> This is likely due to a bug in the wiki parser in our Pig UDF. We can afford
> to lose the data this mapper is working on if it would allow the job to
> finish. Question: is there a hadoop configuration parameter similar to
> mapred.skip.map.max.skip.records that would let us skip a map that doesn't
> progress after X amount of time? Any other possible workarounds for this
> case would also be useful.
>
> We are currently using hadoop 1.1.0 and Pig 0.10.1.
>
> Thanks,
> Chris

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB