Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> RE: Task process exit with nonzero status of 1


+
Marc Limotte 2009-10-27, 21:48
+
Marc Limotte 2009-09-23, 18:06
+
Edward Capriolo 2009-09-24, 14:50
Copy link to this message
-
RE: Task process exit with nonzero status of 1
Thanks for the suggestion, Edward. I only upgraded the JVM after the problem occurred to see if it would help, but it made no difference.

Marc

-----Original Message-----
From: Edward Capriolo [mailto:[EMAIL PROTECTED]]
Sent: Thursday, September 24, 2009 7:50 AM
To: [EMAIL PROTECTED]
Subject: Re: Task process exit with nonzero status of 1

On Wed, Sep 23, 2009 at 2:06 PM, Marc Limotte <[EMAIL PROTECTED]> wrote:
> I'm seeing this error when I try to run my job.
>
> java.io.IOException: Task process exit with nonzero status of 1.
>    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)
>
> From what I can find by doing some Google searches, this means the mapred task JVM has crashed.  Not many suggestions about what to do about it.  Some suggestions about increasing max heap.  I tried that, although I don't think that's the issue because it's not a particularly memory intensive process and I've even tried it with a super small input data set of only a few records.  Still see the same issue.
>
> Can't find anything else in the logs.  I don't think my task even started, because there are no user logs created at all. Seems to fail during Job Setup.
>
> A little more background.  This job was working fine for weeks, running hourly, and then failed on Saturday morning and hasn't worked since.  Obviously, I looked for something that changed at that point, but no one was working at that time... can't find anything that changed.  I tried the job with different input data sets, doesn't seem to matter, unless I run it with no data at all.  The job does run with no input data, but if I have even a few input records it fails-doesn't seem to matter which records.  I suspected some corruption in HDFS, but I was able to extract the data from HDFS (hadoop dfs -get ...) and the data looks ok.  I also copied this data set to our TEST cluster and ran the job there... and it WORKED!
>
> Ran one of our other jobs and it failed as well, so it doesn't seem to be job specific either; looks like every job fails the same way.
>
> Did a complete reboot of the cluster-no impact.
>
> We're using Hadoop 0.20.0, and Java 1.6 update 16 on CentOS 5.2 64bit.
>
> Any suggestions on what could be wrong or where to look for more information would be appreciated.
>
>
>
> Marc Limotte
> Feeva Technology
>
> PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
>
Just a shot in the dark....

Did you update java recently

http://www.koopman.me/2009/04/hadoop-0183-could-not-create-the-java-virtual-machine/

PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
+
Todd Lipcon 2009-09-24, 17:18
+
Marc Limotte 2009-09-24, 18:24
+
Feng, Ao 2009-10-09, 17:47
+
Frank Singleton 2009-10-09, 18:27
+
Todd Lipcon 2009-09-24, 18:27
+
Marc Limotte 2009-09-24, 21:19
+
Marc Limotte 2009-09-24, 22:54
+
Vinod KV 2009-09-25, 03:22
+
Koji Noguchi 2009-09-24, 18:37
+
Marc Limotte 2009-09-24, 21:11