Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Mapper always hangs at the same spot


+
Harsh J 2013-04-13, 16:00
Copy link to this message
-
Re: Mapper always hangs at the same spot
Edward Capriolo 2013-04-13, 20:08
Your application logic is likely stuck in a loop.
On Sat, Apr 13, 2013 at 12:47 PM, Chris Hokamp <[EMAIL PROTECTED]>wrote:

> >When you say "never progresses", do you see the MR framework kill it
> >automatically after 10 minutes of inactivity or does it never ever
> >exit?
>
> The latter -- it never exits. Killing it manually seems like a good option
> for now. We already have mapred.max.map.failures.percent set to a
> non-zero value, but because the task never fails, this never comes into
> effect.
>
> Thanks for the help,
> Chris
>
>
> On Sat, Apr 13, 2013 at 5:00 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> When you say "never progresses", do you see the MR framework kill it
>> automatically after 10 minutes of inactivity or does it never ever
>> exit?
>>
>> You can lower the timeout period on tasks via mapred.task.timeout set
>> in msec. You could also set mapred.max.map.failures.percent to a
>> non-zero value to allow that much percentage of tasks to fail without
>> also marking the whole job as a failure.
>>
>> If the task itself does not get killed by the framework due to
>> inactiveness, try doing a hadoop job -fail-task on its attempt ID
>> manually.
>>
>> On Sat, Apr 13, 2013 at 8:45 PM, Chris Hokamp <[EMAIL PROTECTED]>
>> wrote:
>> > Hello,
>> >
>> > We have a job where all mappers finish except for one, which always
>> hangs at
>> > the same spot (i.e. reaches 49%, then never progresses).
>> >
>> > This is likely due to a bug in the wiki parser in our Pig UDF. We can
>> afford
>> > to lose the data this mapper is working on if it would allow the job to
>> > finish. Question: is there a hadoop configuration parameter similar to
>> > mapred.skip.map.max.skip.records that would let us skip a map that
>> doesn't
>> > progress after X amount of time? Any other possible workarounds for this
>> > case would also be useful.
>> >
>> > We are currently using hadoop 1.1.0 and Pig 0.10.1.
>> >
>> > Thanks,
>> > Chris
>>
>>
>>
>> --
>> Harsh J
>>
>
>
+
Azuryy Yu 2013-04-14, 00:20