Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - Too many open files (java.net.SocketException)

Copy link to this message
Too many open files (java.net.SocketException)
Jean-Marc Spaggiari 2013-04-09, 15:33

I just faced an issue this morning on one of my RS.

Here is an extract of the logs
2013-04-09 11:05:33,164 ERROR org.apache.hadoop.hdfs.DFSClient:
Exception closing file
: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:323)
at sun.nio.ch.Net.socket(Net.java:316)
at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:101)
at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3423)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3381)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)

ulimit is unlimited on all my servers.

Is seems there was to many network connections opened. Is there
anything HBase can handle in such scenario? It's only hadoop in the
stacktrace, so I'm not sure.

Can this be related to nproc? I don't think so. I have another tool
running on the RS. Using low CPU, low bandwidth but making MANY
network HTTP connections...

Any suggestion?

Ted Yu 2013-04-09, 16:00
Jean-Marc Spaggiari 2013-04-09, 16:08
Andrew Purtell 2013-04-09, 16:36
Ted 2013-04-10, 00:53