Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%


Copy link to this message
-
RE: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%
You don't need to explicitly call the initTasks(), you can find this invocation in JobTracker.java class.

>*Yet the **Cannot create task split for <my-job-id>** is still there*.
>About errors relating to the JobID, is this significant? --- INFO
>org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
>nReduces=1 max=-1

Here it shows max=-1, which means max configured(i.e default) tasks for the Job is -1.

I think you can see this exception from JT log.

LOG.info(jobId + ": nMaps=" + numMapTasks + " nReduces=" + numReduceTasks + " max=" + maxTasks);
    if (maxTasks > 0 && (numMapTasks + numReduceTasks) > maxTasks) {
      throw new IOException(
                "The number of tasks for this job " +
                (numMapTasks + numReduceTasks) +
                " exceeds the configured limit " + maxTasks);
    }

Can you try configuring some number for this configuration 'mapred.jobtracker.maxtasks.per.job' and run the same Job?
Thanks
Devaraj k
-----Original Message-----
From: Sreejith Ramakrishnan [mailto:[EMAIL PROTECTED]]
Sent: 11 July 2013 13:26
To: [EMAIL PROTECTED]
Subject: Re: My Job always in 'SUBMIT' state. Stuck at Map: 0% - Reduce 0%

@Devaraj K

*I tried explicitly calling incomingJob.initTasks()*and it gave me a pleasant INFO org.apache.hadoop.mapred.JobInProgress: Job
job_201307111248_0002 initialized successfully with 1 map tasks and 1 reduce tasks

*Yet the **Cannot create task split for <my-job-id>** is still there*.
About errors relating to the JobID, is this significant? --- INFO
org.apache.hadoop.mapred.JobInProgress: job_201307111248_0002: nMaps=1
nReduces=1 max=-1

*P.S: In the log file I shared, anything with "sreejith == " were logged by me. Not system-generated*
On Thu, Jul 11, 2013 at 1:18 PM, Sreejith Ramakrishnan < [EMAIL PROTECTED]> wrote:

> I've attached the relevant portion of the log.
>
> For easy testing, *I've attached the maven project* for the
> EDFJobScheduler. Just make a JAR from it and put it in
> $HADOOP_HOME/lib and add this property to *mapred-site.xml*:
>
> *<property>
> *
> *    <name>mapred.jobtracker.taskScheduler</name>
>     <value>org.apache.hadoop.mapred.EDFJobScheduler</value>
> *
> *</property>*
>
>
> On Thu, Jul 11, 2013 at 12:19 PM, Sreejith Ramakrishnan <
> [EMAIL PROTECTED]> wrote:
>
>> I'm making my own scheduler. Since I had errors, I rewrote the
>> scheduler to do just a simple thing. When I give it a job, it
>> allocates 1 map() and 1
>> reduce() to it. But that isn't happening. I used a barebones WordCount pgm.
>> I'm stuck at map: 0% reduce 0%. And in the jobtracker logs, I see:
>>
>> INFO org.apache.hadoop.mapred.JobInProgress: Cannot create task split
>> for <my-job-id>
>>
>> DETAILED INFO:
>>
>> Here's some snippets of code. The scheduler logic is inside the
>> assignTasks():
>>
>> We get a JobInProgressListener in the constructor-
>> this.jobQueueJobInProgressListener = new
>> JobQueueJobInProgressListener();
>>
>> In start() -
>>
>> this.taskTrackerManager.addJobInProgressListener(jobQueueJobInProgres
>> sListener);
>>
>> Then I just take the last element in that queue (I assume its the
>> latest submitted job) and assign to incomingJob (of type JobInProgress).
>>
>> Then I obtain a new map task as follows and append this task to a
>> list which is the return value of the assignTasks():
>> Task createdMap = incomingJob.obtainNewMapTask(taskTrackerStatus,
>> numTaskTrackers, taskTrackerManager.getNumberOfUniqueHosts());
>>
>>
>> I've attached the scheduler. Can any expert point out if there's any
>> ignorant mistakes from my side?
>>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB