Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Reduce task hang[EMERGENCE]


Copy link to this message
-
Re: Reduce task hang[EMERGENCE]
Hi again,

I did not mention the TaskTracker log, but the actual Task JVM's
syslog/stderr/stdout. Do those log *any* problems at all?

On Fri, Jan 3, 2014 at 11:16 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
> Hi Harsh,
> Thanks.
>
> There is no any error logs for attempt_201312201200_34795_r_000000_0 in the
> tasktracker log. only '0.0% reduce > copy >'
>
> I configured all hosts in all slaves and master.
>
> This job has only one reduce. it hanged. but I configured everybody's max
> job running to '1' in the Fair scheduler file.
>
> but some people's max job running greater than one. and these people's job
> never hanged...
>
>
> On Fri, Jan 3, 2014 at 1:13 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Does the Reduce task log (of attempt_201312201200_34795_r_000000_0)
>> show any errors in trying to communicate with the various TaskTrackers
>> in trying to obtain the data?
>>
>> On Fri, Jan 3, 2014 at 9:54 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>> > Add addtional:
>> >
>> > Our MR version is 1.2.1, not 1.0.4
>> >
>> > There is no useful information in the JT log.
>> >
>> >
>> > On Fri, Jan 3, 2014 at 12:20 PM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Our prod cluster met some issues recently,
>> >> All map tasks finished successfully, but reduce task hanged.
>> >>
>> >> but It's not happened on all TaskTrackers, only sometimes. we used
>> >> mapred-1.0.4
>> >>
>> >> There is "0.0% reduce > copy >" forever until kill task manually.
>> >>
>> >> reduce logs on the TaskTracker:
>> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:13:57 INFO
>> >> mapred.TaskTracker: JVM with ID: jvm_201312201200_34795_r_-365330778
>> >> given
>> >> task: attempt_201312201200_34795_r_000000_0
>> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:04 INFO
>> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0% reduce >
>> >> copy
>> >> >
>> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:08 INFO
>> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0% reduce >
>> >> copy
>> >> >
>> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:14 INFO
>> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0% reduce >
>> >> copy
>> >> >
>> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:17 INFO
>> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0% reduce >
>> >> copy
>> >> >
>> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:23 INFO
>> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0% reduce >
>> >> copy
>> >> >
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>

--
Harsh J