Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> DFSClient error


Copy link to this message
-
Re: DFSClient error
Thanks for the quick response, appreciate it. It looks like this might be
the issue. But I am still trying to understand what is causing so many
threads in my situation? Is this thread per block that gets created or per
file? Because if it's per file then it should not be more than 15.
My second question, I read around 5 .gz files in 5 separate processed. This
is constant and also the size of those 5 is roughly equivalent. So then why
does it fail only halfway and not right in the begining. I am reading
around 400 files and it always fails when I reach around 180th file.

What's the default value of xceivers? Is 4096 consume too much of stack
size?

Thanks
On Sun, Apr 29, 2012 at 1:14 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> It sounds to me like you're running out of DN xceivers. Try the
> solution offered at
> http://hbase.apache.org/book.html#dfs.datanode.max.xcievers
>
> I.e., add:
>
> <property>
>        <name>dfs.datanode.max.xcievers</name>
>        <value>4096</value>
>  </property>
>
> To your DNs' config/hdfs-site.xml and restart the DNs.
>
> On Mon, Apr 30, 2012 at 1:35 AM, Mohit Anchlia <[EMAIL PROTECTED]>
> wrote:
> > I even tried to lower number of parallel jobs even further but I still
> get
> > these errors. Any suggestion on how to troubleshoot this issue would be
> > very helpful. Should I run hadoop fsck? How do people troubleshoot such
> > issues?? Does it sound like a bug?
> >
> > 2012-04-27 14:37:42,921 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 1 map-reduce job(s) waiting for submission.
> > 2012-04-27 14:37:42,931 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Exception in createBlockOutputStream 125.18.62.199:50010
> java.io.EOFException
> > 2012-04-27 14:37:42,932 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Abandoning block blk_6343044536824463287_24619
> > 2012-04-27 14:37:42,932 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Excluding datanode 125.18.62.199:50010
> > 2012-04-27 14:37:42,935 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Exception in createBlockOutputStream 125.18.62.204:50010
> java.io.EOFException
> > 2012-04-27 14:37:42,935 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Abandoning block blk_2837215798109471362_24620
> > 2012-04-27 14:37:42,936 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Excluding datanode 125.18.62.204:50010
> > 2012-04-27 14:37:42,937 [main] INFO
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 1 map-reduce job(s) waiting for submission.
> > 2012-04-27 14:37:42,939 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Exception in createBlockOutputStream 125.18.62.198:50010
> java.io.EOFException
> > 2012-04-27 14:37:42,939 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Abandoning block blk_2223489090936415027_24620
> > 2012-04-27 14:37:42,940 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Excluding datanode 125.18.62.198:50010
> > 2012-04-27 14:37:42,943 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Exception in createBlockOutputStream 125.18.62.197:50010
> java.io.EOFException
> > 2012-04-27 14:37:42,943 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Abandoning block blk_1265169201875643059_24620
> > 2012-04-27 14:37:42,944 [Thread-5] INFO
>  org.apache.hadoop.hdfs.DFSClient -
> > Excluding datanode 125.18.62.197:50010
> > 2012-04-27 14:37:42,945 [Thread-5] WARN
>  org.apache.hadoop.hdfs.DFSClient -
> > DataStreamer Exception: java.io.IOException: Unable to create new block.
> >        at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3446)
> >        at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2627)
> >        at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2822)
> > 2012-04-27 14:37:42,945 [Thread-5] WARN
>  org.apache.hadoop.hdfs.DFSClient -