Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> RE: Task process exit with nonzero status of 1


+
Marc Limotte 2009-10-27, 21:48
+
Marc Limotte 2009-09-23, 18:06
+
Edward Capriolo 2009-09-24, 14:50
+
Marc Limotte 2009-09-24, 16:57
+
Todd Lipcon 2009-09-24, 17:18
+
Marc Limotte 2009-09-24, 18:24
+
Feng, Ao 2009-10-09, 17:47
+
Frank Singleton 2009-10-09, 18:27
+
Todd Lipcon 2009-09-24, 18:27
+
Marc Limotte 2009-09-24, 21:19
Copy link to this message
-
RE: Task process exit with nonzero status of 1
One more clue.

If I change "mapred.job.tracker" to "local" on this cluster, then the I can run the job successfully.  I guess in this case it doesn't have to launch the child JVM, which is the thing that is failing.
Marc

-----Original Message-----
From: Marc Limotte [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 24, 2009 2:19 PM
To: [EMAIL PROTECTED]
Cc: Deept Kumar
Subject: RE: Task process exit with nonzero status of 1

Added DEBUG, but don't see anything interesting. The only new tasktracker log entries are about receiving a heartbeat from the JobTracker, or about cleaning up the task afterward.

Tried the strace. It produces over 6mm lines of output. Not sure what I should be looking for.

I'm thinking I might try the Cloudera Hadoop 0.20.0 distribution and see if the behavior is any different.

Marc

-----Original Message-----
From: Todd Lipcon [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 24, 2009 11:28 AM
To: [EMAIL PROTECTED]
Subject: Re: Task process exit with nonzero status of 1

Odd...

Try bumping up the logs to debug level on that tasktracker, see what you can
determine?

You could also strace -f -p <tasktracker pid> -o /tmp/tt_log and then grep
through those logs later to see what might be going on.

-Todd

On Thu, Sep 24, 2009 at 11:24 AM, Marc Limotte <[EMAIL PROTECTED]> wrote:

> Hi Todd.
>
> No userlogs seem to be created.  I'm guessing, because the map task never
> actually starts.
>
> I don't see any other errors in the tasktracker log, other than the one I
> put in the first message ("java.io.IOException: Task process exit with
> nonzero status of 1...").  I've included the output from one of the nodes'
> tasktracker logs below.
>
> Any other suggestions?
>
> Marc
>
> 2009-09-24 18:15:36,955 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_m_000003_0 task's
> state:UNASSIGNED
> 2009-09-24 18:15:36,959 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> to launch : attempt_200909221656_0006_m_000003_0
> 2009-09-24 18:15:36,960 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_m_000003_02009-09-24 18:15:37,483 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_m_-145
> 18051982009-09-24 18:15:37,483 INFO org.apache.hadoop.mapred.JvmManager:
> JVM Runner jvm_200909221656_0006_m_-1451805198 spawned.
> 2009-09-24 18:15:37,511 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_m_-1451805198 exited. Number of t
> asks it ran: 02009-09-24 18:15:37,512 WARN
> org.apache.hadoop.mapred.TaskRunner: attempt_200909221656_0006_m_000003_0
> Child Error
> java.io.IOException: Task process exit with nonzero status of 1.
>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
> 2009-09-24 18:15:40,518 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_m_000003_0 done; removing files.
> 2009-09-24 18:15:40,519 INFO org.apache.hadoop.mapred.TaskTracker:
> addFreeSlot : current free slots : 2
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_r
> _000001_0 task's state:UNASSIGNED2009-09-24 18:15:42,964 INFO
> org.apache.hadoop.mapred.TaskTracker: Trying to launch :
> attempt_200909221656_0006_r_000001_0
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_r_000001_02009-09-24 18:15:43,000 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_r_7885
> 020722009-09-24 18:15:43,000 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_200909221656_0006_r_788502072 spawned.
> 2009-09-24 18:15:43,026 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_r_788502072 exited. Number of tas
> ks it ran: 0

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
+
Vinod KV 2009-09-25, 03:22
+
Koji Noguchi 2009-09-24, 18:37
+
Marc Limotte 2009-09-24, 21:11