Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%


Copy link to this message
-
Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
Devaraj k 2013-07-11, 13:18
Could you debug and check the Job Tracker/your scheduler code, then you can
pinpoint the exact issue and fix it.

Thanks
Devaraj k
On Thu, Jul 11, 2013 at 7:58 PM, Sreejith Ramakrishnan <
[EMAIL PROTECTED]> wrote:

> @Devaraj K
>
> I tried setting mapred.jobtracker.maxtasks.per.job to 4 in mapred-site.xml.
> In the subsequent job, I got this in the log which clearly shows the
> configuration has taken effect:
>
> INFO org.apache.hadoop.mapred.JobInProgress: job_201307111721_0001: nMaps=1
> nReduces=1 max=4
>
> Yet, the original problem remains. It doesn't go anywhere from 0%. Btw,
> when we get the jobQueue, should we specify a name for the queue we want?
> (eg. default)
>
> INFO org.apache.hadoop.mapred.JobTracker: Job job_201307111721_0001 added
> successfully for user 'sreejith' to queue 'default'
>
> Also, is there a limit to the no.of tasks which can be obtained in a
> heartbeat?
>
>
> On Thu, Jul 11, 2013 at 2:15 PM, Devaraj k <[EMAIL PROTECTED]> wrote:
>
> > You don't need to explicitly call the initTasks(), you can find this
> > invocation in JobTracker.java class.
> >
> > >*Yet the **Cannot create task split for <my-job-id>** is still there*.
> > >About errors relating to the JobID, is this significant? --- INFO
> > >org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> > >nReduces=1 max=-1
> >
> > Here it shows max=-1, which means max configured(i.e default) tasks for
> > the Job is -1.
> >
> > I think you can see this exception from JT log.
> >
> > LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks
> > + " max=" + maxTasks);
> >     if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
> >       throw new IOException(
> >                 "The number of tasks for this job " +
> >                 (numMapTasks + numReduceTasks) +
> >                 " exceeds the configured limit " + maxTasks);
> >     }
> >
> > Can you try configuring some number for this configuration
> > 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
> >
> >
> > Thanks
> > Devaraj k
> >
> >
> > -----Original Message-----
> > From: Sreejith Ramakrishnan [mailto:[EMAIL PROTECTED]]
> > Sent: 11 July 2013 13:26
> > To: [EMAIL PROTECTED]
> > Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce
> 0%
> >
> > @Devaraj K
> >
> > *I tried explicitly calling incomingJob.initTasks()*and it gave me a
> > pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
> > job_201307111248_0002 initialized successfully with 1 map tasks and 1
> > reduce tasks
> >
> > *Yet the **Cannot create task split for <my-job-id>** is still there*.
> > About errors relating to the JobID, is this significant? --- INFO
> > org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
> > nReduces=1 max=-1
> >
> > *P.S: In the log file I shared, anything with "sreejith == " were logged
> > by me. Not system-generated*
> >
> >
> > On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan <
> > [EMAIL PROTECTED]> wrote:
> >
> > > I've attached the relevant portion of the log.
> > >
> > > For easy testing, *I've attached the maven project* for the
> > > EDFJobScheduler. Just make a JAR from it and put it in
> > > $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
> > >
> > > *<property>
> > > *
> > > *    <name>mapred.jobtracker.taskScheduler</name>
> > >     <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> > > *
> > > *</property>*
> > >
> > >
> > > On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > >> I'm making my own scheduler. Since I had errors, I rewrote the
> > >> scheduler to do just a simple thing. When I give it a job, it
> > >> allocates 1 map() and 1
> > >> reduce() to it. But that isn't happening. I used a barebones WordCount
> > pgm.
> > >> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
> > >>
> > >> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split