Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%


Copy link to this message
-
RE: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
You don't need to explicitly call the initTasks(), you can find this invocation in JobTracker.java class.

>*Yet the **Cannot create task split for <my-job-id>** is still there*.
>About errors relating to the JobID, is this significant? --- INFO
>org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
>nReduces=1 max=-1

Here it shows max=-1, which means max configured(i.e default) tasks for the Job is -1.

I think you can see this exception from JT log.

LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks + " max=" + maxTasks);
    if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
      throw new IOException(
                "The number of tasks for this job " +
                (numMapTasks + numReduceTasks) +
                " exceeds the configured limit " + maxTasks);
    }

Can you try configuring some number for this configuration 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
Thanks
Devaraj k
-----Original Message-----
From: Sreejith Ramakrishnan [mailto:[EMAIL PROTECTED]]
Sent: 11 July 2013 13:26
To: [EMAIL PROTECTED]
Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%

@Devaraj K

*I tried explicitly calling incomingJob.initTasks()*and it gave me a pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201307111248_0002 initialized successfully with 1 map tasks and 1 reduce tasks

*Yet the **Cannot create task split for <my-job-id>** is still there*.
About errors relating to the JobID, is this significant? --- INFO
org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
nReduces=1 max=-1

*P.S: In the log file I shared, anything with "sreejith == " were logged by me. Not system-generated*
On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan < [EMAIL PROTECTED]> wrote:

> I've attached the relevant portion of the log.
>
> For easy testing, *I've attached the maven project* for the
> EDFJobScheduler. Just make a JAR from it and put it in
> $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
>
> *<property>
> *
> *    <name>mapred.jobtracker.taskScheduler</name>
>     <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> *
> *</property>*
>
>
> On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> [EMAIL PROTECTED]> wrote:
>
>> I'm making my own scheduler. Since I had errors, I rewrote the
>> scheduler to do just a simple thing. When I give it a job, it
>> allocates 1 map() and 1
>> reduce() to it. But that isn't happening. I used a barebones WordCount pgm.
>> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
>>
>> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split
>> for <my-job-id>
>>
>> DETAILED INFO:
>>
>> Here's some snippets of code. The scheduler logic is inside the
>> assignTasks():
>>
>> We get a JobInProgressListener in the constructor-
>> this.jobQueueJobInProgressListener = new
>> JobQueueJobInProgressListener();
>>
>> In start() -
>>
>> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
>> sListener);
>>
>> Then I just take the last element in that queue (I assume its the
>> latest submitted job) and assign to incomingJob (of type JobInProgress).
>>
>> Then I obtain a new map task as follows and append this task to a
>> list which is the return value of the assignTasks():
>> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
>> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
>>
>>
>> I've attached the scheduler. Can any expert point out if there's any
>> ignorant mistakes from my side?
>>
>
>