Hello,

I saw this error when testing my patch on Jenkins:

Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at java.nio.channels.Selector.open(Selector.java:209)
at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
at sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
... 55 more

(Full log: https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/)

I am not sure who maintains the Jenkins server, but it would be nice to
increase the maximum number of open files for HBase unit tests.

Thanks,
--Mikhail

Mikhail:
If you look at
https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you would
see:

max memory size         (kbytes, -m) unlimited
open files                      (-n) 32768

It is not clear why 32768 was not enough for TestAdmin.

Normally we don't regard test failure resulted from 'Too many open files'
as real test failure.

I agree the effective limit should be raised.

Cheers

On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
[EMAIL PROTECTED]> wrote:

> Hello,
>
> I saw this error when testing my patch on Jenkins:
>
> Caused by: java.io.IOException: Too many open files
>        at sun.nio.ch.IOUtil.initPipe(Native Method)
>        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>        at
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>        at java.nio.channels.Selector.open(Selector.java:209)
>        at
> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
>        at sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
> Source)
>        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at java.lang.Class.newInstance0(Class.java:355)
>        at java.lang.Class.newInstance(Class.java:308)
>        at
> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
>        ... 55 more
>
> (Full log:
> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
> )
>
> I am not sure who maintains the Jenkins server, but it would be nice to
> increase the maximum number of open files for HBase unit tests.
>
> Thanks,
> --Mikhail
>

I disagree that we should raise the limit. If we're using up 32k
descriptors in unit tests, it's probably a real leak we should be
concerned about. Maybe we can add a temporary patch upstream which
does:
catch (IOException ioe) {
  if (ioe.getMessage().contains("Too many open files")) {
    Runtime.getRuntime().exec("lsof -u " + System.getProperty("user.name"));
  }
}
or something like that?

-Todd

On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Mikhail:
> If you look at
> https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you would
> see:
>
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 32768
>
> It is not clear why 32768 was not enough for TestAdmin.
>
> Normally we don't regard test failure resulted from 'Too many open files'
> as real test failure.
>
> I agree the effective limit should be raised.
>
> Cheers
>
> On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
> [EMAIL PROTECTED]> wrote:
>
>> Hello,
>>
>> I saw this error when testing my patch on Jenkins:
>>
>> Caused by: java.io.IOException: Too many open files
>>        at sun.nio.ch.IOUtil.initPipe(Native Method)
>>        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>        at
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>        at java.nio.channels.Selector.open(Selector.java:209)
>>        at
>> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
>>        at sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
>> Source)
>>        at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>        at java.lang.Class.newInstance0(Class.java:355)
>>        at java.lang.Class.newInstance(Class.java:308)
>>        at
>> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
>>        ... 55 more
>>
>> (Full log:
>> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
>> )
>>
>> I am not sure who maintains the Jenkins server, but it would be nice to
>> increase the maximum number of open files for HBase unit tests.
>>
>> Thanks,
>> --Mikhail
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

I agree. The tests should not need more than 32k open files.

---From: Todd Lipcon <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc:
Sent: Saturday, November 26, 2011 10:42 AM
Subject: Re: max number of open files when testing HBase on the Jenkins server

I disagree that we should raise the limit. If we're using up 32k
descriptors in unit tests, it's probably a real leak we should be
concerned about. Maybe we can add a temporary patch upstream which
does:
catch (IOException ioe) {
  if (ioe.getMessage().contains("Too many open files")) {
    Runtime.getRuntime().exec("lsof -u " + System.getProperty("user.name"));
  }
}
or something like that?

-Todd

On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Mikhail:
> If you look at
> https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you would
> see:
>
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 32768
>
> It is not clear why 32768 was not enough for TestAdmin.
>
> Normally we don't regard test failure resulted from 'Too many open files'
> as real test failure.
>
> I agree the effective limit should be raised.
>
> Cheers
>
> On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
> [EMAIL PROTECTED]> wrote:
>
>> Hello,
>>
>> I saw this error when testing my patch on Jenkins:
>>
>> Caused by: java.io.IOException: Too many open files
>>        at sun.nio.ch.IOUtil.initPipe(Native Method)
>>        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>>        at
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>>        at java.nio.channels.Selector.open(Selector.java:209)
>>        at
>> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
>>        at sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
>> Source)
>>        at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>>        at java.lang.Class.newInstance0(Class.java:355)
>>        at java.lang.Class.newInstance(Class.java:308)
>>        at
>> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
>>        ... 55 more
>>
>> (Full log:
>> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
>> )
>>
>> I am not sure who maintains the Jenkins server, but it would be nice to
>> increase the maximum number of open files for HBase unit tests.
>>
>> Thanks,
>> --Mikhail
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

We can observe the difference between Jenkins build setup and PreCommit
build setup here:
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK/2487/console

Linux hemera 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC
2011 x86_64 GNU/Linux
[HBase-TRUNK] $ /bin/bash -xe /tmp/hudson4606140560635170338.sh
+ ulimit -a

...

open files                      (-n) 40000

We have many issues to tackle, build failure being one of them.

Trying to reduce number of descriptors to 32k could be a goal for the
future. For the time being, its priority is not high.

Cheers

On Sat, Nov 26, 2011 at 5:01 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> I agree. The tests should not need more than 32k open files.
>
>
>
> ----- Original Message -----
> From: Todd Lipcon <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Saturday, November 26, 2011 10:42 AM
> Subject: Re: max number of open files when testing HBase on the Jenkins
> server
>
> I disagree that we should raise the limit. If we're using up 32k
> descriptors in unit tests, it's probably a real leak we should be
> concerned about. Maybe we can add a temporary patch upstream which
> does:
> catch (IOException ioe) {
>   if (ioe.getMessage().contains("Too many open files")) {
>     Runtime.getRuntime().exec("lsof -u " + System.getProperty("user.name
> "));
>   }
> }
> or something like that?
>
> -Todd
>
> On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Mikhail:
> > If you look at
> > https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you
> would
> > see:
> >
> > max memory size         (kbytes, -m) unlimited
> > open files                      (-n) 32768
> >
> > It is not clear why 32768 was not enough for TestAdmin.
> >
> > Normally we don't regard test failure resulted from 'Too many open files'
> > as real test failure.
> >
> > I agree the effective limit should be raised.
> >
> > Cheers
> >
> > On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Hello,
> >>
> >> I saw this error when testing my patch on Jenkins:
> >>
> >> Caused by: java.io.IOException: Too many open files
> >>        at sun.nio.ch.IOUtil.initPipe(Native Method)
> >>        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >>        at
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >>        at java.nio.channels.Selector.open(Selector.java:209)
> >>        at
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
> >>        at sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
> >> Source)
> >>        at
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >>        at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >>        at java.lang.Class.newInstance0(Class.java:355)
> >>        at java.lang.Class.newInstance(Class.java:308)
> >>        at
> >> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
> >>        ... 55 more
> >>
> >> (Full log:
> >>
> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
> >> )
> >>
> >> I am not sure who maintains the Jenkins server, but it would be nice to
> >> increase the maximum number of open files for HBase unit tests.
> >>
> >> Thanks,
> >> --Mikhail
> >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>

On Sun, Nov 27, 2011 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> We have many issues to tackle, build failure being one of them.
>
> Trying to reduce number of descriptors to 32k could be a goal for the
> future. For the time being, its priority is not high.

My feeling is that this is a bug - the build failure is doing exactly
what it's supposed to: alerting us of a bug. If the builds are using
this many FDs, it's likely a leak -- just like the "out of memory"
error last week was actually do to an infinite loop spewing exceptions
in a test.

Avoiding these errors by raising limits just hides the bug so we find
it later in production. Unless we have a good explanation that shows
that it's "just a test issue" we shouldn't ignore it or hide it by
raising ulimit.

-Todd

>
> On Sat, Nov 26, 2011 at 5:01 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
>> I agree. The tests should not need more than 32k open files.
>>
>>
>>
>> ----- Original Message -----
>> From: Todd Lipcon <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Cc:
>> Sent: Saturday, November 26, 2011 10:42 AM
>> Subject: Re: max number of open files when testing HBase on the Jenkins
>> server
>>
>> I disagree that we should raise the limit. If we're using up 32k
>> descriptors in unit tests, it's probably a real leak we should be
>> concerned about. Maybe we can add a temporary patch upstream which
>> does:
>> catch (IOException ioe) {
>>   if (ioe.getMessage().contains("Too many open files")) {
>>     Runtime.getRuntime().exec("lsof -u " + System.getProperty("user.name
>> "));
>>   }
>> }
>> or something like that?
>>
>> -Todd
>>
>> On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > Mikhail:
>> > If you look at
>> > https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you
>> would
>> > see:
>> >
>> > max memory size         (kbytes, -m) unlimited
>> > open files                      (-n) 32768
>> >
>> > It is not clear why 32768 was not enough for TestAdmin.
>> >
>> > Normally we don't regard test failure resulted from 'Too many open files'
>> > as real test failure.
>> >
>> > I agree the effective limit should be raised.
>> >
>> > Cheers
>> >
>> > On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> Hello,
>> >>
>> >> I saw this error when testing my patch on Jenkins:
>> >>
>> >> Caused by: java.io.IOException: Too many open files
>> >>        at sun.nio.ch.IOUtil.initPipe(Native Method)
>> >>        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
>> >>        at
>> >>
>> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
>> >>        at java.nio.channels.Selector.open(Selector.java:209)
>> >>        at
>> >>
>> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
>> >>        at sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
>> >> Source)
>> >>        at
>> >>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>> >>        at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>> >>        at java.lang.Class.newInstance0(Class.java:355)
>> >>        at java.lang.Class.newInstance(Class.java:308)
>> >>        at
>> >> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
>> >>        ... 55 more
>> >>
>> >> (Full log:
>> >>
>> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
>> >> )
>> >>
>> >> I am not sure who maintains the Jenkins server, but it would be nice to
>> >> increase the maximum number of open files for HBase unit tests.
>> >>
>> >> Thanks,
>> >> --Mikhail
>> >>
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

Looks like hunting down FD leak would be a good candidate for Tuesday's
Hackathon :)

On Sun, Nov 27, 2011 at 9:56 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Sun, Nov 27, 2011 at 9:29 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > We have many issues to tackle, build failure being one of them.
> >
> > Trying to reduce number of descriptors to 32k could be a goal for the
> > future. For the time being, its priority is not high.
>
> My feeling is that this is a bug - the build failure is doing exactly
> what it's supposed to: alerting us of a bug. If the builds are using
> this many FDs, it's likely a leak -- just like the "out of memory"
> error last week was actually do to an infinite loop spewing exceptions
> in a test.
>
> Avoiding these errors by raising limits just hides the bug so we find
> it later in production. Unless we have a good explanation that shows
> that it's "just a test issue" we shouldn't ignore it or hide it by
> raising ulimit.
>
> -Todd
>
> >
> > On Sat, Nov 26, 2011 at 5:01 PM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
> >
> >> I agree. The tests should not need more than 32k open files.
> >>
> >>
> >>
> >> ----- Original Message -----
> >> From: Todd Lipcon <[EMAIL PROTECTED]>
> >> To: [EMAIL PROTECTED]
> >> Cc:
> >> Sent: Saturday, November 26, 2011 10:42 AM
> >> Subject: Re: max number of open files when testing HBase on the Jenkins
> >> server
> >>
> >> I disagree that we should raise the limit. If we're using up 32k
> >> descriptors in unit tests, it's probably a real leak we should be
> >> concerned about. Maybe we can add a temporary patch upstream which
> >> does:
> >> catch (IOException ioe) {
> >>   if (ioe.getMessage().contains("Too many open files")) {
> >>     Runtime.getRuntime().exec("lsof -u " + System.getProperty("
> user.name
> >> "));
> >>   }
> >> }
> >> or something like that?
> >>
> >> -Todd
> >>
> >> On Thu, Nov 24, 2011 at 6:27 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >> > Mikhail:
> >> > If you look at
> >> > https://builds.apache.org/job/PreCommit-HBASE-Build/364/console, you
> >> would
> >> > see:
> >> >
> >> > max memory size         (kbytes, -m) unlimited
> >> > open files                      (-n) 32768
> >> >
> >> > It is not clear why 32768 was not enough for TestAdmin.
> >> >
> >> > Normally we don't regard test failure resulted from 'Too many open
> files'
> >> > as real test failure.
> >> >
> >> > I agree the effective limit should be raised.
> >> >
> >> > Cheers
> >> >
> >> > On Thu, Nov 24, 2011 at 5:03 PM, Mikhail Bautin <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> Hello,
> >> >>
> >> >> I saw this error when testing my patch on Jenkins:
> >> >>
> >> >> Caused by: java.io.IOException: Too many open files
> >> >>        at sun.nio.ch.IOUtil.initPipe(Native Method)
> >> >>        at
> sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:49)
> >> >>        at
> >> >>
> >>
> sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
> >> >>        at java.nio.channels.Selector.open(Selector.java:209)
> >> >>        at
> >> >>
> >>
> org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:42)
> >> >>        at
> sun.reflect.GeneratedConstructorAccessor40.newInstance(Unknown
> >> >> Source)
> >> >>        at
> >> >>
> >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >> >>        at
> >> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >> >>        at java.lang.Class.newInstance0(Class.java:355)
> >> >>        at java.lang.Class.newInstance(Class.java:308)
> >> >>        at
> >> >>
> org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1737)
> >> >>        ... 55 more
> >> >>
> >> >> (Full log:
> >> >>
> >>
> https://builds.apache.org/job/PreCommit-HBASE-Build/364//testReport/org.apache.hadoop.hbase.client/TestAdmin/testCheckHBaseAvailableClosesConnection/
> >> >> )
> >> >>
> >> >> I am not sure who maintains the Jenkins server, but it would be nice