anthony garnier
2011-12-23, 14:32
alo alt
2011-12-23, 14:40
anthony garnier
2011-12-27, 10:49
anthony garnier
2011-12-27, 12:20
anthony garnier
2011-12-27, 13:54
alo alt
2011-12-27, 14:07
anthony garnier
2011-12-27, 14:18
anthony garnier
2011-12-27, 16:18
-
Task process exit with nonzero status of 134anthony garnier 2011-12-23, 14:32
Hi folks,
I've just done a fresh install of Hadoop, Namenode and datanode are up, Task/job Tracker also up, but when I run the Map reduce worcount exemple I got this error on Task tracker:
2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of tasks it ran: 0
2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112231511_0001_m_000002_0 : Child Error
java.io.IOException: Task process exit with nonzero status of 134.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
And job tracker stuck :
# hadoop jar hadoop-examples*.jar wordcount input/test.txt output/
11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to process : 1
11/12/23 15:11:49 INFO mapred.JobClient: Running job: job_201112231511_0001
11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0%
I'running hadoop 0.20.203.0, java 1.6.0 rev 25
I've done some googling, apparently the JVM crash hard (maybe Out of memory), does someone have any hint ?
Regards,
Anthony Garnier
/DSIN/ASTI/ETSO
IT Center
PSA Peugeot Citroen
Bessoncourt 90160
-
Re: Task process exit with nonzero status of 134alo alt 2011-12-23, 14:40
Hi,
take a look into the logs for the failed attempt at your Tasktracker.
Also check the system logs with dmesg or /var/log/kern*. Could be a
syskill (segfault).
- Alex
On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
>
> Hi folks,
>
> I've just done a fresh install of Hadoop, Namenode and datanode are up, Task/job Tracker also up, but when I run the Map reduce worcount exemple I got this error on Task tracker:
>
> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of tasks it ran: 0
> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112231511_0001_m_000002_0 : Child Error
> java.io.IOException: Task process exit with nonzero status of 134.
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
>
> And job tracker stuck :
> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/
> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to process : 1
> 11/12/23 15:11:49 INFO mapred.JobClient: Running job: job_201112231511_0001
> 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0%
>
>
> I'running hadoop 0.20.203.0, java 1.6.0 rev 25
>
> I've done some googling, apparently the JVM crash hard (maybe Out of memory), does someone have any hint ?
>
> Regards,
>
> Anthony Garnier
> /DSIN/ASTI/ETSO
> IT Center
> PSA Peugeot Citroen
> Bessoncourt 90160
>
>
>
>
--
Alexander Lorenz
http://mapredit.blogspot.com
P Think of the environment: please don't print this email unless you
really need to.
take a look into the logs for the failed attempt at your Tasktracker.
Also check the system logs with dmesg or /var/log/kern*. Could be a
syskill (segfault).
- Alex
On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
>
> Hi folks,
>
> I've just done a fresh install of Hadoop, Namenode and datanode are up, Task/job Tracker also up, but when I run the Map reduce worcount exemple I got this error on Task tracker:
>
> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of tasks it ran: 0
> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112231511_0001_m_000002_0 : Child Error
> java.io.IOException: Task process exit with nonzero status of 134.
> at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
>
> And job tracker stuck :
> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/
> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to process : 1
> 11/12/23 15:11:49 INFO mapred.JobClient: Running job: job_201112231511_0001
> 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0%
>
>
> I'running hadoop 0.20.203.0, java 1.6.0 rev 25
>
> I've done some googling, apparently the JVM crash hard (maybe Out of memory), does someone have any hint ?
>
> Regards,
>
> Anthony Garnier
> /DSIN/ASTI/ETSO
> IT Center
> PSA Peugeot Citroen
> Bessoncourt 90160
>
>
>
>
--
Alexander Lorenz
http://mapredit.blogspot.com
P Think of the environment: please don't print this email unless you
really need to.
-
Re: Task process exit with nonzero status of 134anthony garnier 2011-12-27, 10:49
Hi,
I got Nothing in the dmesg
I've checked the Tasktracker and this is what I got :
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.203.0
STARTUP_MSG: build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
************************************************************/
2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2011-12-23 15:11:02,330 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2011-12-23 15:11:02,331 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics system started
2011-12-23 15:11:02,597 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: Starting tasktracker with owner as root
2011-12-23 15:11:02,870 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2011-12-23 15:11:02,871 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered.
2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort58709 registered.
2011-12-23 15:11:02,900 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort58709 registered.
2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 58709: starting
2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 58709: starting
2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 58709: starting
2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 58709: starting
2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: TaskTracker up at: localhost/127.0.0.1:58709
2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: Starting tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 58709: starting
2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker: Using MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1
2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker: Starting thread: Map-events fetcher for all reduce tasks on tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
2011-12-23 15:11:02,964 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2011-12-23 15:11:02,965 WARN org.apache.hadoop.mapred.TaskTracker: TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled.
2011-12-23 15:11:02,970 INFO org.apache.hadoop.mapred.IndexCache: IndexCache created with max memory = 10485760
2011-12-23 15:11:02,976 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ShuffleServerMetrics registered.
2011-12-23 15:11:02,978 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50060
2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() returned 50060
2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
2011-12-23 15:11:02,979 INFO org.mortbay.log: jetty-6.1.26
2011-12-23 15:11:03,257 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50060
2011-12-23 15:11:03,257 INFO org.apache.hadoop.mapred.TaskTracker: FILE_CACHE_SIZE for mapOutputServlet set to : 2000
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231432_0002 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231121_0001 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231312_0002 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231142_0002 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112230933_0002 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231357_0001 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231052_0002 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201112231426_0001 for user-log deletion with retainTimeStamp:1324735862967
2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerT
-
RE: Task process exit with nonzero status of 134anthony garnier 2011-12-27, 12:20
Alex,
Memory available on namenode / Jobtracker :
Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 15360M total, 11609M used, 3750M free, 311M buffers
Swap: 2047M total, 1M used, 2046M free, 8833M cached
On datanode / Tasktracker :
top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26
Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 72373M total, 4321M used, 68051M free, 348M buffers
Swap: 2047M total, 0M used, 2047M free, 2771M cached
src/mapred/mapred-default.xml :
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx200m</value>
</property>
So there should be enough memory
Anthony
> Date: Tue, 27 Dec 2011 11:58:46 +0100
> Subject: Re: Task process exit with nonzero status of 134
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> Anthony,
>
> How much memory you have available? Did the system going into swap?
>
> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx).
> - what says "top -Hc"?
>
> - Alex
>
> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I got Nothing in the dmesg
> > I've checked the Tasktracker and this is what I got :
> >
> > /************************************************************
> > STARTUP_MSG: Starting TaskTracker
> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.203.0
> > STARTUP_MSG: build > > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
> > ************************************************************/
> > 2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
> > loaded properties from hadoop-metrics2.properties
> > 2011-12-23 15:11:02,330 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > MetricsSystem,sub=Stats registered.
> > 2011-12-23 15:11:02,331 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> > at 10 second(s).
> > 2011-12-23 15:11:02,331 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> > system started
> > 2011-12-23 15:11:02,597 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> > registered.
> > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added global
> > filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> > 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> > 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> > tasktracker with owner as root
> > 2011-12-23 15:11:02,870 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> > registered.
> > 2011-12-23 15:11:02,871 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > TaskTrackerMetrics registered.
> > 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting
> > SocketReader
> > 2011-12-23 15:11:02,900 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcDetailedActivityForPort58709 registered.
> > 2011-12-23 15:11:02,900 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcActivityForPort58709 registered.
> > 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
-
RE: Task process exit with nonzero status of 134anthony garnier 2011-12-27, 13:54
Alex
-XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions)
Anyway, I tested it but same result.
Is it wise to test hadoop with the new jdk7_2 ?
Anthony
> Date: Tue, 27 Dec 2011 13:47:03 +0100
> Subject: Re: Task process exit with nonzero status of 134
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> Anthony,
>
> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest
> update, could be a bug inside. Can you test SE 7u2?
> Todd mentioned in a older post to use -XX:+UseCompressedOops
> (hadoop-env.sh). Another option could be to take a closer look at
> garbage collection with compressed option.
>
> - Alex
>
> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
> > Alex,
> >
> > Memory available on namenode / Jobtracker :
> > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> > 0.0%st
> > Mem: 15360M total, 11609M used, 3750M free, 311M buffers
> > Swap: 2047M total, 1M used, 2046M free, 8833M cached
> >
> > On datanode / Tasktracker :
> > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26
> > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si,
> > 0.0%st
> > Mem: 72373M total, 4321M used, 68051M free, 348M buffers
> > Swap: 2047M total, 0M used, 2047M free, 2771M cached
> >
> > src/mapred/mapred-default.xml :
> > <property>
> > <name>mapred.child.java.opts</name>
> > <value>-Xmx200m</value>
> > </property>
> >
> >
> > So there should be enough memory
> >
> > Anthony
> >
> >
> >> Date: Tue, 27 Dec 2011 11:58:46 +0100
> >
> >> Subject: Re: Task process exit with nonzero status of 134
> >> From: [EMAIL PROTECTED]
> >> To: [EMAIL PROTECTED]
> >
> >>
> >> Anthony,
> >>
> >> How much memory you have available? Did the system going into swap?
> >>
> >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx).
> >> - what says "top -Hc"?
> >>
> >> - Alex
> >>
> >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi,
> >> >
> >> > I got Nothing in the dmesg
> >> > I've checked the Tasktracker and this is what I got :
> >> >
> >> > /************************************************************
> >> > STARTUP_MSG: Starting TaskTracker
> >> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86
> >> > STARTUP_MSG: args = []
> >> > STARTUP_MSG: version = 0.20.203.0
> >> > STARTUP_MSG: build > >> >
> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
> >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
> >> > ************************************************************/
> >> > 2011-12-23 15:11:02,275 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsConfig:
> >> > loaded properties from hadoop-metrics2.properties
> >> > 2011-12-23 15:11:02,330 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > MetricsSystem,sub=Stats registered.
> >> > 2011-12-23 15:11:02,331 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> >> > period
> >> > at 10 second(s).
> >> > 2011-12-23 15:11:02,331 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> >> > system started
> >> > 2011-12-23 15:11:02,597 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > ugi
> >> > registered.
> >> > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to
> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> >> > org.mortbay.log.Slf4jLog
> >> > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added
> >> > global
> >> > filtersafety
> >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
-
Re: Task process exit with nonzero status of 134alo alt 2011-12-27, 14:07
Anthony,
personally I haven't tested yet, jdk7 have bugs already. Was only a
hint to see if the error occurs.
I would focus on memory issues, the installed RAM are okay? No errors?
My next step would be to downgrade into one JDK earlier to check for a
bug. Did you update the OS before?
- Alex
On Tue, Dec 27, 2011 at 2:54 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
>
> Alex
>
> -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions)
> Anyway, I tested it but same result.
> Is it wise to test hadoop with the new jdk7_2 ?
>
> Anthony
>
>
>> Date: Tue, 27 Dec 2011 13:47:03 +0100
>> Subject: Re: Task process exit with nonzero status of 134
>> From: [EMAIL PROTECTED]
>
>> To: [EMAIL PROTECTED]
>>
>> Anthony,
>>
>> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest
>> update, could be a bug inside. Can you test SE 7u2?
>> Todd mentioned in a older post to use -XX:+UseCompressedOops
>> (hadoop-env.sh). Another option could be to take a closer look at
>> garbage collection with compressed option.
>>
>> - Alex
>>
>> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
>> > Alex,
>> >
>> > Memory available on namenode / Jobtracker :
>> > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie
>> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
>> > 0.0%st
>> > Mem: 15360M total, 11609M used, 3750M free, 311M buffers
>> > Swap: 2047M total, 1M used, 2046M free, 8833M cached
>> >
>> > On datanode / Tasktracker :
>> > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26
>> > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie
>> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si,
>> > 0.0%st
>> > Mem: 72373M total, 4321M used, 68051M free, 348M buffers
>> > Swap: 2047M total, 0M used, 2047M free, 2771M cached
>> >
>> > src/mapred/mapred-default.xml :
>> > <property>
>> > <name>mapred.child.java.opts</name>
>> > <value>-Xmx200m</value>
>> > </property>
>> >
>> >
>> > So there should be enough memory
>> >
>> > Anthony
>> >
>> >
>> >> Date: Tue, 27 Dec 2011 11:58:46 +0100
>> >
>> >> Subject: Re: Task process exit with nonzero status of 134
>> >> From: [EMAIL PROTECTED]
>> >> To: [EMAIL PROTECTED]
>> >
>> >>
>> >> Anthony,
>> >>
>> >> How much memory you have available? Did the system going into swap?
>> >>
>> >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx).
>> >> - what says "top -Hc"?
>> >>
>> >> - Alex
>> >>
>> >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > I got Nothing in the dmesg
>> >> > I've checked the Tasktracker and this is what I got :
>> >> >
>> >> > /************************************************************
>> >> > STARTUP_MSG: Starting TaskTracker
>> >> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86
>> >> > STARTUP_MSG: args = []
>> >> > STARTUP_MSG: version = 0.20.203.0
>> >> > STARTUP_MSG: build >> >> >
>> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
>> >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
>> >> > ************************************************************/
>> >> > 2011-12-23 15:11:02,275 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsConfig:
>> >> > loaded properties from hadoop-metrics2.properties
>> >> > 2011-12-23 15:11:02,330 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> >> > MetricsSystem,sub=Stats registered.
>> >> > 2011-12-23 15:11:02,331 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> >> > period
>> >> > at 10 second(s).
>> >> > 2011-12-23 15:11:02,331 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>
Alexander Lorenz
http://mapredit.blogspot.com
P Think of the environment: please don't print this email unless you
really need to.
personally I haven't tested yet, jdk7 have bugs already. Was only a
hint to see if the error occurs.
I would focus on memory issues, the installed RAM are okay? No errors?
My next step would be to downgrade into one JDK earlier to check for a
bug. Did you update the OS before?
- Alex
On Tue, Dec 27, 2011 at 2:54 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
>
> Alex
>
> -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions)
> Anyway, I tested it but same result.
> Is it wise to test hadoop with the new jdk7_2 ?
>
> Anthony
>
>
>> Date: Tue, 27 Dec 2011 13:47:03 +0100
>> Subject: Re: Task process exit with nonzero status of 134
>> From: [EMAIL PROTECTED]
>
>> To: [EMAIL PROTECTED]
>>
>> Anthony,
>>
>> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest
>> update, could be a bug inside. Can you test SE 7u2?
>> Todd mentioned in a older post to use -XX:+UseCompressedOops
>> (hadoop-env.sh). Another option could be to take a closer look at
>> garbage collection with compressed option.
>>
>> - Alex
>>
>> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
>> > Alex,
>> >
>> > Memory available on namenode / Jobtracker :
>> > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie
>> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
>> > 0.0%st
>> > Mem: 15360M total, 11609M used, 3750M free, 311M buffers
>> > Swap: 2047M total, 1M used, 2046M free, 8833M cached
>> >
>> > On datanode / Tasktracker :
>> > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26
>> > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie
>> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si,
>> > 0.0%st
>> > Mem: 72373M total, 4321M used, 68051M free, 348M buffers
>> > Swap: 2047M total, 0M used, 2047M free, 2771M cached
>> >
>> > src/mapred/mapred-default.xml :
>> > <property>
>> > <name>mapred.child.java.opts</name>
>> > <value>-Xmx200m</value>
>> > </property>
>> >
>> >
>> > So there should be enough memory
>> >
>> > Anthony
>> >
>> >
>> >> Date: Tue, 27 Dec 2011 11:58:46 +0100
>> >
>> >> Subject: Re: Task process exit with nonzero status of 134
>> >> From: [EMAIL PROTECTED]
>> >> To: [EMAIL PROTECTED]
>> >
>> >>
>> >> Anthony,
>> >>
>> >> How much memory you have available? Did the system going into swap?
>> >>
>> >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx).
>> >> - what says "top -Hc"?
>> >>
>> >> - Alex
>> >>
>> >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > I got Nothing in the dmesg
>> >> > I've checked the Tasktracker and this is what I got :
>> >> >
>> >> > /************************************************************
>> >> > STARTUP_MSG: Starting TaskTracker
>> >> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86
>> >> > STARTUP_MSG: args = []
>> >> > STARTUP_MSG: version = 0.20.203.0
>> >> > STARTUP_MSG: build >> >> >
>> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
>> >> > -r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
>> >> > ************************************************************/
>> >> > 2011-12-23 15:11:02,275 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsConfig:
>> >> > loaded properties from hadoop-metrics2.properties
>> >> > 2011-12-23 15:11:02,330 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
>> >> > MetricsSystem,sub=Stats registered.
>> >> > 2011-12-23 15:11:02,331 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
>> >> > period
>> >> > at 10 second(s).
>> >> > 2011-12-23 15:11:02,331 INFO
>> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
>
Alexander Lorenz
http://mapredit.blogspot.com
P Think of the environment: please don't print this email unless you
really need to.
-
RE: Task process exit with nonzero status of 134anthony garnier 2011-12-27, 14:18
Alex
I don't know why but now I got other error code :
On Jobtracker :
java.lang.Throwable: Child Error
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
On taskTracker :
2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's launchTask follows:
2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.TaskController:
2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201112271514_0001_m_-1820714542 but just removed
2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112271514_0001_m_-1820714542 exited with exit code 1. Number of tasks it ran: 0
2011-12-27 15:14:50,666 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112271514_0001_m_000002_0 : Child Error
java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
2011-12-27 15:14:53,676 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: RE: Task process exit with nonzero status of 134
Date: Tue, 27 Dec 2011 13:54:22 +0000
Alex
-XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions)
Anyway, I tested it but same result.
Is it wise to test hadoop with the new jdk7_2 ?
Anthony
> Date: Tue, 27 Dec 2011 13:47:03 +0100
> Subject: Re: Task process exit with nonzero status of 134
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> Anthony,
>
> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest
> update, could be a bug inside. Can you test SE 7u2?
> Todd mentioned in a older post to use -XX:+UseCompressedOops
> (hadoop-env.sh). Another option could be to take a closer look at
> garbage collection with compressed option.
>
> - Alex
>
> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
> > Alex,
> >
> > Memory available on namenode / Jobtracker :
> > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> > 0.0%st
> > Mem: 15360M total, 11609M used, 3750M free, 311M buffers
> > Swap: 2047M total, 1M used, 2046M free, 8833M cached
> >
> > On datanode / Tasktracker :
> > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28, 0.26
> > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si,
> > 0.0%st
> > Mem: 72373M total, 4321M used, 68051M free, 348M buffers
> > Swap: 2047M total, 0M used, 2047M free, 2771M cached
> >
> > src/mapred/mapred-default.xml :
> > <property>
> > <name>mapred.child.java.opts</name>
> > <value>-Xmx200m</value>
> > </property>
> >
> >
> > So there should be enough memory
> >
> > Anthony
> >
> >
> >> Date: Tue, 27 Dec 2011 11:58:46 +0100
> >
> >> Subject: Re: Task process exit with nonzero status of 134
> >> From: [EMAIL PROTECTED]
> >> To: [EMAIL PROTECTED]
> >
> >>
> >> Anthony,
> >>
> >> How much memory you have available? Did the system going into swap?
> >>
> >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx).
> >> - what says "top -Hc"?
> >>
> >> - Alex
> >>
> >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi,
> >> >
> >> > I got Nothing in the dmesg
> >> > I've checked the Tasktracker and this is what I got :
> >> >
> >> > /************************************************************
> >> > STARTUP_MSG: Starting TaskTracker
> >> > STARTUP_MSG: host = ylal2960.inetpsa.com/10.68.217.86
-
RE: Task process exit with nonzero status of 134anthony garnier 2011-12-27, 16:18
Alex
java -version -Xmx1024m is running fine
I was also able to create directory in /users/hadoop-0.20.203.0/log/userlogs
But anyway I've doe a fresh install with the 0.20.205.0 version and the mapreduce job completed successfully !
Thank you for your help.
Anthony
> Date: Tue, 27 Dec 2011 17:06:28 +0100
> Subject: Re: Task process exit with nonzero status of 134
> From: [EMAIL PROTECTED]
> To: [EMAIL PROTECTED]
>
> Anthoney,
>
> That could be when you try to allocate some dedicated space and it is
> not available or present on your machine.
> Did -Xmx has an dedicated memory value that you can provide to JVM on
> your machine (e.g. "-Xmx1024m")?
>
> Second - filesystem full (mostly hadoop/userlogs). Ext3 has an limit
> of 32k links per node. Try to create a directory in (mkdir
> hadoop/userlogs/123), if you have too much directories an error will
> occur.
>
> - Alex
>
>
>
>
>
> On Tue, Dec 27, 2011 at 3:18 PM, anthony garnier <[EMAIL PROTECTED]> wrote:
> > Alex
> > I don't know why but now I got other error code :
> >
> > On Jobtracker :
> > java.lang.Throwable: Child Error
> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
> > Caused by: java.io.IOException: Task process exit with nonzero status of 1.
> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> >
> > On taskTracker :
> >
> > 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.DefaultTaskController:
> > Output from DefaultTaskController's launchTask follows:
> > 2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.TaskController:
> > 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM Not
> > killed jvm_201112271514_0001_m_-1820714542 but just removed
> > 2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201112271514_0001_m_-1820714542 exited with exit code 1. Number of tasks
> > it ran: 0
> > 2011-12-27 15:14:50,666 WARN org.apache.hadoop.mapred.TaskRunner:
> > attempt_201112271514_0001_m_000002_0 : Child Error
> > java.io.IOException: Task process exit with nonzero status of 1.
> > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> > 2011-12-27 15:14:53,676 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> >
> >
> > ________________________________
> > From: [EMAIL PROTECTED]
> > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > Subject: RE: Task process exit with nonzero status of 134
> > Date: Tue, 27 Dec 2011 13:54:22 +0000
> >
> >
> > Alex
> >
> > -XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit
> > JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions)
> > Anyway, I tested it but same result.
> > Is it wise to test hadoop with the new jdk7_2 ?
> >
> > Anthony
> >
> >
> >> Date: Tue, 27 Dec 2011 13:47:03 +0100
> >> Subject: Re: Task process exit with nonzero status of 134
> >> From: [EMAIL PROTECTED]
> >
> >> To: [EMAIL PROTECTED]
> >>
> >> Anthony,
> >>
> >> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest
> >> update, could be a bug inside. Can you test SE 7u2?
> >> Todd mentioned in a older post to use -XX:+UseCompressedOops
> >> (hadoop-env.sh). Another option could be to take a closer look at
> >> garbage collection with compressed option.
> >>
> >> - Alex
> >>
> >> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <[EMAIL PROTECTED]>
> >> wrote:
> >> > Alex,
> >> >
> >> > Memory available on namenode / Jobtracker :
> >> > Tasks: 435 total, 1 running, 434 sleeping, 0 stopped, 0 zombie
> >> > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
> >> > 0.0%st
> >> > Mem: 15360M total, 11609M used, 3750M free, 311M buffers
> >> > Swap: 2047M total, 1M used, 2046M free, 8833M cached
> >> >
> >> > On datanode / Tasktracker :
> >> > top - 13:15:27 up 6 days, 21:11, 1 user, load average: 0.03, 0.28,
> >> > 0.26
> >> > Tasks: 377 total, 1 running, 376 sleeping, 0 stopped, 0 zombie
