Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?

Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=59462] throws
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012 11:01:28 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.lang.Thread.run(Thread.java:636)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.lang.Thread.start(Thread.java:614)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at java.lang.Thread.start0(Native Method)
Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: java.lang.OutOfMemoryError: unable to create new native thread

Not known specifically that I know of. Can you give us more details? Did
you actually run out of memory? File descriptors? What does a heap dump
tell you?

C

On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
[EMAIL PROTECTED]> wrote:

> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
> Accept-0: accept loop for ServerSocket[addr> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012 11:01:28
> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.lang.Thread.run(Thread.java:636)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.lang.Thread.start(Thread.java:614)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> java.lang.Thread.start0(Native Method)
> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
> java.lang.OutOfMemoryError: unable to create new native thread
>
>

It's not really out of memory. It's "unable to create new native thread". It appears to have to do with JMX (notice the RMI).

-JZ

On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:

> Not known specifically that I know of. Can you give us more details? Did
> you actually run out of memory? File descriptors? What does a heap dump
> tell you?
>
> C
>
> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
> [EMAIL PROTECTED]> wrote:
>
>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>> Accept-0: accept loop for ServerSocket[addr>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012 11:01:28
>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.lang.Thread.run(Thread.java:636)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.lang.Thread.start(Thread.java:614)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>> java.lang.Thread.start0(Native Method)
>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>> java.lang.OutOfMemoryError: unable to create new native thread
>>
>>

Yeah a tickle in the back of my mind says that this is a JVM bug. When was
the last time you restarted these instances?

C

On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
[EMAIL PROTECTED]> wrote:

> It's not really out of memory. It's "unable to create new native thread".
> It appears to have to do with JMX (notice the RMI).
>
> -JZ
>
> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>
> > Not known specifically that I know of. Can you give us more details? Did
> > you actually run out of memory? File descriptors? What does a heap dump
> > tell you?
> >
> > C
> >
> > On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
> >>
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
> >> Accept-0: accept loop for ServerSocket[addr> >> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
> 11:01:28
> >> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> java.lang.Thread.run(Thread.java:636)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> java.lang.Thread.start(Thread.java:614)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >> java.lang.Thread.start0(Native Method)
> >> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
> >> java.lang.OutOfMemoryError: unable to create new native thread
> >>
> >>
>
>

I'm not sure which of the instances went down, but it could be a few weeks.

-JZ

On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:

> Yeah a tickle in the back of my mind says that this is a JVM bug. When was
> the last time you restarted these instances?
>
> C
>
> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
> [EMAIL PROTECTED]> wrote:
>
>> It's not really out of memory. It's "unable to create new native thread".
>> It appears to have to do with JMX (notice the RMI).
>>
>> -JZ
>>
>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>
>>> Not known specifically that I know of. Can you give us more details? Did
>>> you actually run out of memory? File descriptors? What does a heap dump
>>> tell you?
>>>
>>> C
>>>
>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>> Accept-0: accept loop for ServerSocket[addr>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>> 11:01:28
>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> java.lang.Thread.run(Thread.java:636)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> java.lang.Thread.start(Thread.java:614)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>> java.lang.Thread.start0(Native Method)
>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>
>>>>
>>
>>

What does a jstack on another long-running instance show? Anything out of
the norm?

C

On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
[EMAIL PROTECTED]> wrote:

> I'm not sure which of the instances went down, but it could be a few weeks.
>
> -JZ
>
> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>
> > Yeah a tickle in the back of my mind says that this is a JVM bug. When
> was
> > the last time you restarted these instances?
> >
> > C
> >
> > On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
> > [EMAIL PROTECTED]> wrote:
> >
> >> It's not really out of memory. It's "unable to create new native
> thread".
> >> It appears to have to do with JMX (notice the RMI).
> >>
> >> -JZ
> >>
> >> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
> >>
> >>> Not known specifically that I know of. Can you give us more details?
> Did
> >>> you actually run out of memory? File descriptors? What does a heap dump
> >>> tell you?
> >>>
> >>> C
> >>>
> >>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
> >>> [EMAIL PROTECTED]> wrote:
> >>>
> >>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
> >>>>
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
> >>>> Accept-0: accept loop for ServerSocket[addr> >>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
> >> 11:01:28
> >>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>> java.lang.Thread.run(Thread.java:636)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> >>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>>
> >>
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>> java.lang.Thread.start(Thread.java:614)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
> >>>> java.lang.Thread.start0(Native Method)
> >>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
> >>>> java.lang.OutOfMemoryError: unable to create new native thread
> >>>>
> >>>>
> >>
> >>
>
>

Good idea - I'll try that.

-JZ

On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:

> What does a jstack on another long-running instance show? Anything out of
> the norm?
>
> C
>
> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
> [EMAIL PROTECTED]> wrote:
>
>> I'm not sure which of the instances went down, but it could be a few weeks.
>>
>> -JZ
>>
>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>
>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>> was
>>> the last time you restarted these instances?
>>>
>>> C
>>>
>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> It's not really out of memory. It's "unable to create new native
>> thread".
>>>> It appears to have to do with JMX (notice the RMI).
>>>>
>>>> -JZ
>>>>
>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>
>>>>> Not known specifically that I know of. Can you give us more details?
>> Did
>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>> tell you?
>>>>>
>>>>> C
>>>>>
>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>> Accept-0: accept loop for ServerSocket[addr>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>> 11:01:28
>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start0(Native Method)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>

Hi Jordan,

BTW, could you give details of -Xmx, -Xss  parameters?  OS ? 32bit/62bit system?

As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.

Also, there could be chances of thread leak in the system.

-Rakesh
________________________________________
From: Jordan Zimmerman [[EMAIL PROTECTED]]
Sent: Monday, July 09, 2012 8:43 AM
To: [EMAIL PROTECTED]
Subject: Re: OOM/no more threads?

Good idea - I'll try that.

-JZ

On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:

> What does a jstack on another long-running instance show? Anything out of
> the norm?
>
> C
>
> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
> [EMAIL PROTECTED]> wrote:
>
>> I'm not sure which of the instances went down, but it could be a few weeks.
>>
>> -JZ
>>
>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>
>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>> was
>>> the last time you restarted these instances?
>>>
>>> C
>>>
>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> It's not really out of memory. It's "unable to create new native
>> thread".
>>>> It appears to have to do with JMX (notice the RMI).
>>>>
>>>> -JZ
>>>>
>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>
>>>>> Not known specifically that I know of. Can you give us more details?
>> Did
>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>> tell you?
>>>>>
>>>>> C
>>>>>
>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>> [EMAIL PROTECTED]> wrote:
>>>>>
>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>> Accept-0: accept loop for ServerSocket[addr>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>> 11:01:28
>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>
>>>>
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>> java.lang.Thread.start0(Native Method)
>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>

-Xms35004M -Xmx35004M -Xss128k

64 bit Centos

On Jul 8, 2012, at 10:07 PM, Rakesh R wrote:

> Hi Jordan,
>
> BTW, could you give details of -Xmx, -Xss  parameters?  OS ? 32bit/62bit system?
>
> As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.
>
> Also, there could be chances of thread leak in the system.
>
> -Rakesh
> ________________________________________
> From: Jordan Zimmerman [[EMAIL PROTECTED]]
> Sent: Monday, July 09, 2012 8:43 AM
> To: [EMAIL PROTECTED]
> Subject: Re: OOM/no more threads?
>
> Good idea - I'll try that.
>
> -JZ
>
> On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
>
>> What does a jstack on another long-running instance show? Anything out of
>> the norm?
>>
>> C
>>
>> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I'm not sure which of the instances went down, but it could be a few weeks.
>>>
>>> -JZ
>>>
>>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>>
>>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>>> was
>>>> the last time you restarted these instances?
>>>>
>>>> C
>>>>
>>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> It's not really out of memory. It's "unable to create new native
>>> thread".
>>>>> It appears to have to do with JMX (notice the RMI).
>>>>>
>>>>> -JZ
>>>>>
>>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>>
>>>>>> Not known specifically that I know of. Can you give us more details?
>>> Did
>>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>>> tell you?
>>>>>>
>>>>>> C
>>>>>>
>>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>>
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>>> Accept-0: accept loop for ServerSocket[addr>>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>>> 11:01:28
>>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start0(Native Method)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>

1) Probably good to relook the Linux thread stack size. Use “ulimit -s <size in kb>” for setting slightly bigger value and observe the test case.
2) Check also your ulimit on virtual size, i.e. ulimit -v. Increasing this limit should let you create more threads with the same stack size.
3) Also, lowering -Xmx can help by leaving more memory for thread stacks.

All these are like tuning options. Still would be good to observe the jstack once again in long running instance and will see the possibilities of leaks as Camille suggested?

-Rakesh
________________________________________
From: Jordan Zimmerman [[EMAIL PROTECTED]]
Sent: Monday, July 09, 2012 12:57 PM
To: [EMAIL PROTECTED]
Subject: Re: OOM/no more threads?

 -Xms35004M -Xmx35004M -Xss128k

64 bit Centos

On Jul 8, 2012, at 10:07 PM, Rakesh R wrote:

> Hi Jordan,
>
> BTW, could you give details of -Xmx, -Xss  parameters?  OS ? 32bit/62bit system?
>
> As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.
>
> Also, there could be chances of thread leak in the system.
>
> -Rakesh
> ________________________________________
> From: Jordan Zimmerman [[EMAIL PROTECTED]]
> Sent: Monday, July 09, 2012 8:43 AM
> To: [EMAIL PROTECTED]
> Subject: Re: OOM/no more threads?
>
> Good idea - I'll try that.
>
> -JZ
>
> On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
>
>> What does a jstack on another long-running instance show? Anything out of
>> the norm?
>>
>> C
>>
>> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I'm not sure which of the instances went down, but it could be a few weeks.
>>>
>>> -JZ
>>>
>>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>>
>>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>>> was
>>>> the last time you restarted these instances?
>>>>
>>>> C
>>>>
>>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> It's not really out of memory. It's "unable to create new native
>>> thread".
>>>>> It appears to have to do with JMX (notice the RMI).
>>>>>
>>>>> -JZ
>>>>>
>>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>>
>>>>>> Not known specifically that I know of. Can you give us more details?
>>> Did
>>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>>> tell you?
>>>>>>
>>>>>> C
>>>>>>
>>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>>
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>>> Accept-0: accept loop for ServerSocket[addr>>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>>> 11:01:28
>>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start0(Native Method)