Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> socket timeouts, dropped packages


Copy link to this message
-
Re: socket timeouts, dropped packages
Please keep Cloudera issues off this list.

On Apr 13, 2011, at 12:22 PM, Eli Collins wrote:

> Hey Ferdy,
>
> If you're seeing this after bumping fs.datanode.max.xcievers and the
> nfiles ulimit, and you're also seeing dropped packets it sounds like
> you're having networking issues.
>
> See the following as well:
> https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/1d3a377bd605e1bd/d3d8ec0d14c065bb?
> #d3d8ec0d14c065bb
>
> Thanks,
> Eli
>
> On Tue, Apr 12, 2011 at 10:37 AM, Ferdy Galema <[EMAIL PROTECTED]
> > wrote:
>> Hi,
>>
>> We're running into issues were we are seeing timeouts when writing/
>> reading a
>> lot of hdfs data. (Hadoop is version CDH4B3 and hdfs appending is  
>> enabled).
>> The type of exceptions vary a lot, but most of the times it's  
>> whenever a
>> DFSClient writes data into the datanodes pipeline.
>>
>> For example, one datanode logs "Exception in receiveBlock for block
>> blk_5476601577216704980_62953994 java.io.EOFException: while trying  
>> to read
>> 65557 bytes" and the other side logs "writeBlock
>> blk_5476601577216704980_62953994 received exception
>> java.net.SocketTimeoutException: Read timed out". That's it.
>>
>> We cannot seem to determine the exact problem. The read timeout is  
>> default
>> (60 sec). The open files limit and the number of xceivers is upped  
>> a lot. A
>> full GC never takes longer than a second.
>>
>> However, we are seeing a lot of dropped packages on the networking
>> interface. Could these problems be related?
>>
>> Any advice will be helpful.
>>
>> Ferdy.
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB