Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> RE: Task process exit with nonzero status of 1


+
Marc Limotte 2009-10-27, 21:48
+
Marc Limotte 2009-09-23, 18:06
+
Edward Capriolo 2009-09-24, 14:50
+
Marc Limotte 2009-09-24, 16:57
+
Todd Lipcon 2009-09-24, 17:18
+
Marc Limotte 2009-09-24, 18:24
+
Feng, Ao 2009-10-09, 17:47
+
Frank Singleton 2009-10-09, 18:27
+
Todd Lipcon 2009-09-24, 18:27
+
Marc Limotte 2009-09-24, 21:19
+
Marc Limotte 2009-09-24, 22:54
+
Vinod KV 2009-09-25, 03:22
+
Koji Noguchi 2009-09-24, 18:37
Copy link to this message
-
RE: Task process exit with nonzero status of 1
Hi Koji,

Thanks for the suggestion.  We have not set mapred.child.ulimit in our hadoop conf files.  And I verified that it was not set in the logged job.conf.  Don't see any limits set at the OS level, either.

Marc

-----Original Message-----
From: Koji Noguchi [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 24, 2009 11:37 AM
To: [EMAIL PROTECTED]
Subject: Re: Task process exit with nonzero status of 1

> > A little more background.  This job was working fine for weeks, running
> > hourly, and then failed on Saturday morning and hasn't worked since.

Any chance that ulimit (mapred.child.ulimit) got enabled?

Koji
On 9/24/09 11:24 AM, "Marc Limotte" <[EMAIL PROTECTED]> wrote:

> Hi Todd.
>
> No userlogs seem to be created.  I'm guessing, because the map task never
> actually starts.
>
> I don't see any other errors in the tasktracker log, other than the one I put
> in the first message ("java.io.IOException: Task process exit with nonzero
> status of 1...").  I've included the output from one of the nodes' tasktracker
> logs below.
>
> Any other suggestions?
>
> Marc
>
> 2009-09-24 18:15:36,955 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_m_000003_0 task's
> state:UNASSIGNED
> 2009-09-24 18:15:36,959 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
> launch : attempt_200909221656_0006_m_000003_0
> 2009-09-24 18:15:36,960 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_m_000003_02009-09-24 18:15:37,483 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_m_-145
> 18051982009-09-24 18:15:37,483 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_200909221656_0006_m_-1451805198 spawned.
> 2009-09-24 18:15:37,511 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_m_-1451805198 exited. Number of t
> asks it ran: 02009-09-24 18:15:37,512 WARN
> org.apache.hadoop.mapred.TaskRunner: attempt_200909221656_0006_m_000003_0
> Child Error
> java.io.IOException: Task process exit with nonzero status of 1.
>         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
> 2009-09-24 18:15:40,518 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_m_000003_0 done; removing files.
> 2009-09-24 18:15:40,519 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot
> : current free slots : 2
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_r
> _000001_0 task's state:UNASSIGNED2009-09-24 18:15:42,964 INFO
> org.apache.hadoop.mapred.TaskTracker: Trying to launch :
> attempt_200909221656_0006_r_000001_0
> 2009-09-24 18:15:42,964 INFO org.apache.hadoop.mapred.TaskTracker: In
> TaskLauncher, current free slots : 2 and trying to launch
>  attempt_200909221656_0006_r_000001_02009-09-24 18:15:43,000 INFO
> org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID:
> jvm_200909221656_0006_r_7885
> 020722009-09-24 18:15:43,000 INFO org.apache.hadoop.mapred.JvmManager: JVM
> Runner jvm_200909221656_0006_r_788502072 spawned.
> 2009-09-24 18:15:43,026 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> jvm_200909221656_0006_r_788502072 exited. Number of tas
> ks it ran: 0
> 2009-09-24 18:15:43,026 WARN org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_r_000001_0 Child Error
> java.io.IOException: Task process exit with nonzero status of 1.
>         at
> org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)2009-09-24
> 18:15:46,034 INFO org.apache.hadoop.mapred.TaskRunner:
> attempt_200909221656_0006_r_000001_0 done; removing files.
> 2009-09-24 18:15:46,039 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot
> : current free slots : 2
> 2009-09-24 18:16:34,022 INFO org.apache.hadoop.mapred.TaskTracker:
> LaunchTaskAction (registerTask): attempt_200909221656_0006_m
> _000002_1 task's state:UNASSIGNED
PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB