Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> NPE on JobTracker when job count > 70


Copy link to this message
-
NPE on JobTracker when job count > 70
Hi , all

I wrote a pig script and it works fine. After I add more computation to the existing pig script, the hadoop job count exceed 70, and I get this error from job tracker:
2012-08-08 00:35:22,648 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8012, call submitJob(job_201208071030_3456, hdfs://DC-hadoop01:9000/home/hadoop/hdfs/hdfstmp/mapred/staging/hadoop/.staging/job_201208071030_3456, org.apache.hadoop.security.Credentials@4ff3b079) from 10.20.100.34:19728: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3644)
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3611)
        at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeThreads(DFSClient.java:4026)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:4069)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:4004)
        at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1514)
        at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:383)
        at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:339)
        at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1499)
        at org.apache.hadoop.fs.FileSystem.closeAllForUGI(FileSystem.java:289)
        at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:491)
        at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3642)

It seems like the DFSOutputStream the job tracker is trying to closing is already closed, so the streamer field is already null.

But I can not turn off the fs cache, since I'm afraid the leaked DFSOutputstream may cause other problems.
Is there any other solution?

Thanks.
Haitao Yao
[EMAIL PROTECTED]
weibo: @haitao_yao
Skype:  haitao.yao.final

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB