Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> DFSClient error


Copy link to this message
-
Re: DFSClient error
I even tried to lower number of parallel jobs even further but I still get
these errors. Any suggestion on how to troubleshoot this issue would be
very helpful. Should I run hadoop fsck? How do people troubleshoot such
issues?? Does it sound like a bug?

2012-04-27 14:37:42,921 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-04-27 14:37:42,931 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.199:50010java.io.EOFException
2012-04-27 14:37:42,932 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_6343044536824463287_24619
2012-04-27 14:37:42,932 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.199:50010
2012-04-27 14:37:42,935 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.204:50010java.io.EOFException
2012-04-27 14:37:42,935 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_2837215798109471362_24620
2012-04-27 14:37:42,936 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.204:50010
2012-04-27 14:37:42,937 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-04-27 14:37:42,939 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException
2012-04-27 14:37:42,939 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_2223489090936415027_24620
2012-04-27 14:37:42,940 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:42,943 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.197:50010java.io.EOFException
2012-04-27 14:37:42,943 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_1265169201875643059_24620
2012-04-27 14:37:42,944 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.197:50010
2012-04-27 14:37:42,945 [Thread-5] WARN  org.apache.hadoop.hdfs.DFSClient -
DataStreamer Exception: java.io.IOException: Unable to create new block.
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3446)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2627)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2822)
2012-04-27 14:37:42,945 [Thread-5] WARN  org.apache.hadoop.hdfs.DFSClient -
Error Recovery for block blk_1265169201875643059_24620 bad datanode[0]
nodes == null
2012-04-27 14:37:42,945 [Thread-5] WARN  org.apache.hadoop.hdfs.DFSClient -
Could not get block locations. Source file
"/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411/job.jar"
- Aborting...
2012-04-27 14:37:42,945 [Thread-4] INFO  org.apache.hadoop.mapred.JobClient
- Cleaning up the staging area
hdfs://dsdb1:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411
2012-04-27 14:37:42,945 [Thread-4] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:java.io.EOFException
2012-04-27 14:37:42,996 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream
125.18.62.200:50010java.io.IOException: Bad connect ack with
firstBadLink as
125.18.62.198:50010
2012-04-27 14:37:42,996 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-7583284266913502018_24621
2012-04-27 14:37:42,997 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException
2012-04-27 14:37:42,997 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_4207260385919079785_24622
2012-04-27 14:37:42,998 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:43,000 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:43,002 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.197:50010java.io.EOFException
2012-04-27 14:37:43,002 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-2859304645525022496_24624
2012-04-27 14:37:43,003 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.197:50010
2012-04-27 14:37:43,003 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException
2012-04-27 14:37:43,004 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-5091361633954135154_24622
2012-04-27 14:37:43,004 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.199:50010java.io.EOFException
2012-04-27 14:37:43,004 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-1445223397912067500_24624
2012-04-27 14:37:43,005 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:43,005 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.199:50010
2012-04-27 14:37:43,006 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.204:50010java.io.EOFException
2012-04-27 14:37:43,006 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_4137744363907213546_24624
2012-04-27 14:37:43,007 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.204:50010
2012-04-27 14:37:43,008 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.204:50010java.io.EOFException
2012-04-27 14:37:43,008 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_45536925356783765