Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> DFSClient error


Copy link to this message
-
Re: DFSClient error
I even tried to lower number of parallel jobs even further but I still get
these errors. Any suggestion on how to troubleshoot this issue would be
very helpful. Should I run hadoop fsck? How do people troubleshoot such
issues?? Does it sound like a bug?

2012-04-27 14:37:42,921 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-04-27 14:37:42,931 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.199:50010java.io.EOFException
2012-04-27 14:37:42,932 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_6343044536824463287_24619
2012-04-27 14:37:42,932 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.199:50010
2012-04-27 14:37:42,935 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.204:50010java.io.EOFException
2012-04-27 14:37:42,935 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_2837215798109471362_24620
2012-04-27 14:37:42,936 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.204:50010
2012-04-27 14:37:42,937 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-04-27 14:37:42,939 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException
2012-04-27 14:37:42,939 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_2223489090936415027_24620
2012-04-27 14:37:42,940 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:42,943 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.197:50010java.io.EOFException
2012-04-27 14:37:42,943 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_1265169201875643059_24620
2012-04-27 14:37:42,944 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.197:50010
2012-04-27 14:37:42,945 [Thread-5] WARN  org.apache.hadoop.hdfs.DFSClient -
DataStreamer Exception: java.io.IOException: Unable to create new block.
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3446)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2627)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2822)
2012-04-27 14:37:42,945 [Thread-5] WARN  org.apache.hadoop.hdfs.DFSClient -
Error Recovery for block blk_1265169201875643059_24620 bad datanode[0]
nodes == null
2012-04-27 14:37:42,945 [Thread-5] WARN  org.apache.hadoop.hdfs.DFSClient -
Could not get block locations. Source file
"/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411/job.jar"
- Aborting...
2012-04-27 14:37:42,945 [Thread-4] INFO  org.apache.hadoop.mapred.JobClient
- Cleaning up the staging area
hdfs://dsdb1:54310/tmp/hadoop-hadoop/mapred/staging/hadoop/.staging/job_201204261707_0411
2012-04-27 14:37:42,945 [Thread-4] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:java.io.EOFException
2012-04-27 14:37:42,996 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream
125.18.62.200:50010java.io.IOException: Bad connect ack with
firstBadLink as
125.18.62.198:50010
2012-04-27 14:37:42,996 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-7583284266913502018_24621
2012-04-27 14:37:42,997 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException
2012-04-27 14:37:42,997 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_4207260385919079785_24622
2012-04-27 14:37:42,998 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:43,000 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:43,002 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.197:50010java.io.EOFException
2012-04-27 14:37:43,002 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-2859304645525022496_24624
2012-04-27 14:37:43,003 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.197:50010
2012-04-27 14:37:43,003 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.198:50010java.io.EOFException
2012-04-27 14:37:43,004 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-5091361633954135154_24622
2012-04-27 14:37:43,004 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.199:50010java.io.EOFException
2012-04-27 14:37:43,004 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_-1445223397912067500_24624
2012-04-27 14:37:43,005 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.198:50010
2012-04-27 14:37:43,005 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.199:50010
2012-04-27 14:37:43,006 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.204:50010java.io.EOFException
2012-04-27 14:37:43,006 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_4137744363907213546_24624
2012-04-27 14:37:43,007 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Excluding datanode 125.18.62.204:50010
2012-04-27 14:37:43,008 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Exception in createBlockOutputStream 125.18.62.204:50010java.io.EOFException
2012-04-27 14:37:43,008 [Thread-5] INFO  org.apache.hadoop.hdfs.DFSClient -
Abandoning block blk_45536925356783765
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB