Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Too many open files (java.net.SocketException)


Copy link to this message
-
Re: Too many open files (java.net.SocketException)
Andrew Purtell 2013-04-09, 16:36
Correct, nproc has nothing to do with file table issues.

I typically do something like this when setting up a node:

echo "@hadoop soft nofile 65536" >> /etc/security/limits.conf
echo "@hadoop hard nofile 65536" >> /etc/security/limits.conf

where all accounts launching Hadoop daemons are in the 'hadoop' group. Make
sure you've done something like this.

You may need to increase the maximum number of file-handles that the Linux
kernel will allocate overall. Try adding this to an init script:

    sysctl -w fs.file-max=131072

... for example. Or you can add "fs.file-max=131072" into /etc/sysctl.conf.
If you do that, then be sure to execute sysctl -p as root for the change to
take effect.

On Tue, Apr 9, 2013 at 9:08 AM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> But there was not any trace looking like "OutOfMemoryError". nproc
> might has result with this, no? Not a SocketException?
> Anyway, I have increased it to 32768. I will see if I face that again.
>
> Thanks,
>
> JM
>
> 2013/4/9 Ted Yu <[EMAIL PROTECTED]>:
> > According to http://hbase.apache.org/book.html#ulimit , you should
> increase
> > nproc setting.
> >
> > Cheers
> >
> > On Tue, Apr 9, 2013 at 8:33 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]
> >> wrote:
> >
> >> Hi,
> >>
> >> I just faced an issue this morning on one of my RS.
> >>
> >> Here is an extract of the logs
> >> 2013-04-09 11:05:33,164 ERROR org.apache.hadoop.hdfs.DFSClient:
> >> Exception closing file
> >>
> >>
> /hbase/entry_proposed/ae4a5d72d4613728ddbcc5a64262371b/.tmp/ed6a0154ef714cd88faf26061cf248d3
> >> : java.net.SocketException: Too many open files
> >> java.net.SocketException: Too many open files
> >>         at sun.nio.ch.Net.socket0(Native Method)
> >>         at sun.nio.ch.Net.socket(Net.java:323)
> >>         at sun.nio.ch.Net.socket(Net.java:316)
> >>         at
> sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:101)
> >>         at
> >>
> sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
> >>         at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
> >>         at
> >>
> org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3423)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3381)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2589)
> >>         at
> >>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2829)
> >>
> >> ulimit is unlimited on all my servers.
> >>
> >> Is seems there was to many network connections opened. Is there
> >> anything HBase can handle in such scenario? It's only hadoop in the
> >> stacktrace, so I'm not sure.
> >>
> >> Can this be related to nproc? I don't think so. I have another tool
> >> running on the RS. Using low CPU, low bandwidth but making MANY
> >> network HTTP connections...
> >>
> >> Any suggestion?
> >>
> >> JM
> >>
>

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)