Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Reduce task hang[EMERGENCE]


Copy link to this message
-
Re: Reduce task hang[EMERGENCE]
Hi Harsh,
There is only three warnnings in stderr

*stderr logs*

log4j:WARN No appenders could be found for logger
(org.apache.hadoop.mapred.Child).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
for more info.
On Fri, Jan 3, 2014 at 1:59 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi again,
>
> I did not mention the TaskTracker log, but the actual Task JVM's
> syslog/stderr/stdout. Do those log *any* problems at all?
>
> On Fri, Jan 3, 2014 at 11:16 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
> > Hi Harsh,
> > Thanks.
> >
> > There is no any error logs for attempt_201312201200_34795_r_000000_0 in
> the
> > tasktracker log. only '0.0% reduce > copy >'
> >
> > I configured all hosts in all slaves and master.
> >
> > This job has only one reduce. it hanged. but I configured everybody's max
> > job running to '1' in the Fair scheduler file.
> >
> > but some people's max job running greater than one. and these people's
> job
> > never hanged...
> >
> >
> > On Fri, Jan 3, 2014 at 1:13 PM, Harsh J <[EMAIL PROTECTED]> wrote:
> >>
> >> Does the Reduce task log (of attempt_201312201200_34795_r_000000_0)
> >> show any errors in trying to communicate with the various TaskTrackers
> >> in trying to obtain the data?
> >>
> >> On Fri, Jan 3, 2014 at 9:54 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
> >> > Add addtional:
> >> >
> >> > Our MR version is 1.2.1, not 1.0.4
> >> >
> >> > There is no useful information in the JT log.
> >> >
> >> >
> >> > On Fri, Jan 3, 2014 at 12:20 PM, Azuryy Yu <[EMAIL PROTECTED]>
> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> Our prod cluster met some issues recently,
> >> >> All map tasks finished successfully, but reduce task hanged.
> >> >>
> >> >> but It's not happened on all TaskTrackers, only sometimes. we used
> >> >> mapred-1.0.4
> >> >>
> >> >> There is "0.0% reduce > copy >" forever until kill task manually.
> >> >>
> >> >> reduce logs on the TaskTracker:
> >> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:13:57 INFO
> >> >> mapred.TaskTracker: JVM with ID: jvm_201312201200_34795_r_-365330778
> >> >> given
> >> >> task: attempt_201312201200_34795_r_000000_0
> >> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:04 INFO
> >> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0%
> reduce >
> >> >> copy
> >> >> >
> >> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:08 INFO
> >> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0%
> reduce >
> >> >> copy
> >> >> >
> >> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:14 INFO
> >> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0%
> reduce >
> >> >> copy
> >> >> >
> >> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:17 INFO
> >> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0%
> reduce >
> >> >> copy
> >> >> >
> >> >> hadoop-hadoop-tasktracker-10-200-91-186.out:14/01/03 06:14:23 INFO
> >> >> mapred.TaskTracker: attempt_201312201200_34795_r_000000_0 0.0%
> reduce >
> >> >> copy
> >> >> >
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>