Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> RE: Task process exit with nonzero status of 1


+
Marc Limotte 2009-10-27, 21:48
+
Marc Limotte 2009-09-23, 18:06
+
Edward Capriolo 2009-09-24, 14:50
+
Marc Limotte 2009-09-24, 16:57
+
Todd Lipcon 2009-09-24, 17:18
+
Marc Limotte 2009-09-24, 18:24
+
Feng, Ao 2009-10-09, 17:47
+
Frank Singleton 2009-10-09, 18:27
+
Todd Lipcon 2009-09-24, 18:27
+
Marc Limotte 2009-09-24, 21:19
Copy link to this message
-
RE: Task process exit with nonzero status of 1
One more clue.

If I change "mapred.job.tracker" to "local" on this cluster, then the I can run the job successfully.  I guess in this case it doesn't have to launch the child JVM, which is the thing that is failing.
Marc

-----Original Message-----
From: Marc Limotte [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 24, 2009 2:19 PM
To: [EMAIL PROTECTED]
Cc: Deept Kumar
Subject: RE: Task process exit with nonzero status of 1

Added DEBUG, but don't see anything interesting. The only new tasktracker log entries are about receiving a heartbeat from the JobTracker, or about cleaning up the task afterward.

Tried the strace. It produces over 6mm lines of output. Not sure what I should be looking for.

I'm thinking I might try the Cloudera Hadoop 0.20.0 distribution and see if the behavior is any different.

Marc

-----Original Message-----
From: Todd Lipcon [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 24, 2009 11:28 AM
To: [EMAIL PROTECTED]
Subject: Re: Task process exit with nonzero status of 1

Odd...

Try bumping up the logs to debug level on that tasktracker, see what you can
determine?

You could also strace -f -p <tasktracker pid> -o /tmp/tt_log and then grep
through those logs later to see what might be going on.

-Todd

On Thu, Sep 24, 2009 at 11:24 AM, Marc Limotte <[EMAIL PROTECTED]> wrote:

> Hi Todd.
>
> No userlogs seem to be created.  I'm guessing, because the map task never
> actually starts.
>
> I don't see any other errors in the tasktracker log, other than the one I
> put in the first message ("java.io.IOException: Task process exit with
> nonzero status of 1...").  I've included the output from one of the nodes'
> tasktracker logs below.
>
> Any other suggestions?
>
> Marc
>
> 2009-09-24 18:15:36,955 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_m_000003_0 task's
> state:UNASSIGNED
> 2009-09-24 18:15:36,959 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to launch : attempt_200909221656_0006_m_000003_0
> 2009-09-24 18:15:36,960 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_m_000003_02009-09-24 18:15:37,483 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_m_-145
> 18051982009-09-24 18:15:37,483 INFO org.apache.hadoop.mapred.JvmManager:
> JVM Runner jvm_200909221656_0006_m_-1451805198 spawned.
> 2009-09-24 18:15:37,511 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_m_-1451805198 exited. Number of t
> asks it ran: 02009-09-24 18:15:37,512 WARN
> org.apache.hadoop.mapred.TaskRunner: attempt_200909221656_0006_m_000003_0
> Child Error
> java.io.IOException: Task process exit with nonzero status of 1.
>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
> 2009-09-24 18:15:40,518 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_m_000003_0 done; removing files.
> 2009-09-24 18:15:40,519 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_r
> _000001_0 task's state:UNASSIGNED2009-09-24 18:15:42,964 INFO
> org.apache.hadoop.mapred.TaskTracker: Trying to launch :
> attempt_200909221656_0006_r_000001_0
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_r_000001_02009-09-24 18:15:43,000 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_r_7885
> 020722009-09-24 18:15:43,000 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_200909221656_0006_r_788502072 spawned.
> 2009-09-24 18:15:43,026 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_r_788502072 exited. Number of tas
> ks it ran: 0

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
+
Vinod KV 2009-09-25, 03:22
+
Koji Noguchi 2009-09-24, 18:37
+
Marc Limotte 2009-09-24, 21:11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB