Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Task process exit with nonzero status of 1


Copy link to this message
-
RE: Task process exit with nonzero status of 1
Hi Koji,

Thanks for the suggestion.  We have not set mapred.child.ulimit in our hadoop conf files.  And I verified that it was not set in the logged job.conf.  Don't see any limits set at the OS level, either.

Marc

-----Original Message-----
From: Koji Noguchi [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 24, 2009 11:37 AM
To: [EMAIL PROTECTED]
Subject: Re: Task process exit with nonzero status of 1

> > A little more background.  This job was working fine for weeks, running
> > hourly, and then failed on Saturday morning and hasn't worked since.

Any chance that ulimit (mapred.child.ulimit) got enabled?

Koji
On 9/24/09 11:24 AM, "Marc Limotte" <[EMAIL PROTECTED]> wrote:

> Hi Todd.
>
> No userlogs seem to be created.  I'm guessing, because the map task never
> actually starts.
>
> I don't see any other errors in the tasktracker log, other than the one I put
> in the first message ("java.io.IOException: Task process exit with nonzero
> status of 1...").  I've included the output from one of the nodes' tasktracker
> logs below.
>
> Any other suggestions?
>
> Marc
>
> 2009-09-24 18:15:36,955 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_m_000003_0 task's
> state:UNASSIGNED
> 2009-09-24 18:15:36,959 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
> launch : attempt_200909221656_0006_m_000003_0
> 2009-09-24 18:15:36,960 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_m_000003_02009-09-24 18:15:37,483 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_m_-145
> 18051982009-09-24 18:15:37,483 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_200909221656_0006_m_-1451805198 spawned.
> 2009-09-24 18:15:37,511 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_m_-1451805198 exited. Number of t
> asks it ran: 02009-09-24 18:15:37,512 WARN
> org.apache.hadoop.mapred.TaskRunner: attempt_200909221656_0006_m_000003_0
> Child Error
> java.io.IOException: Task process exit with nonzero status of 1.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
> 2009-09-24 18:15:40,518 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_m_000003_0 done; removing files.
> 2009-09-24 18:15:40,519 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot
> : current free slots : 2
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_r
> _000001_0 task's state:UNASSIGNED2009-09-24 18:15:42,964 INFO
> org.apache.hadoop.mapred.TaskTracker: Trying to launch :
> attempt_200909221656_0006_r_000001_0
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_r_000001_02009-09-24 18:15:43,000 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_r_7885
> 020722009-09-24 18:15:43,000 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_200909221656_0006_r_788502072 spawned.
> 2009-09-24 18:15:43,026 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_r_788502072 exited. Number of tas
> ks it ran: 0
> 2009-09-24 18:15:43,026 WARN org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_r_000001_0 Child Error
> java.io.IOException: Task process exit with nonzero status of 1.
>         at
> org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)2009-09-24
> 18:15:46,034 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_r_000001_0 done; removing files.
> 2009-09-24 18:15:46,039 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot
> : current free slots : 2
> 2009-09-24 18:16:34,022 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_m
> _000002_1 task's state:UNASSIGNED
PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.