Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> OOM/no more threads?


Copy link to this message
-
RE: OOM/no more threads?

1) Probably good to relook the Linux thread stack size. Use “ulimit -s <size in kb>” for setting slightly bigger value and observe the test case.
2) Check also your ulimit on virtual size, i.e. ulimit -v. Increasing this limit should let you create more threads with the same stack size.
3) Also, lowering -Xmx can help by leaving more memory for thread stacks.

All these are like tuning options. Still would be good to observe the jstack once again in long running instance and will see the possibilities of leaks as Camille suggested?

-Rakesh
________________________________________
From: Jordan Zimmerman [[EMAIL PROTECTED]]
Sent: Monday, July 09, 2012 12:57 PM
To: [EMAIL PROTECTED]
Subject: Re: OOM/no more threads?

 -Xms35004M -Xmx35004M -Xss128k

64 bit Centos

On Jul 8, 2012, at 10:07 PM, Rakesh R wrote:

> Hi Jordan,
>
> BTW, could you give details of -Xmx, -Xss  parameters?  OS ? 32bit/62bit system?
>
> As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.
>
> Also, there could be chances of thread leak in the system.
>
> -Rakesh
> ________________________________________
> From: Jordan Zimmerman [[EMAIL PROTECTED]]
> Sent: Monday, July 09, 2012 8:43 AM
> To: [EMAIL PROTECTED]
> Subject: Re: OOM/no more threads?
>
> Good idea - I'll try that.
>
> -JZ
>
> On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
>
>> What does a jstack on another long-running instance show? Anything out of
>> the norm?
>>
>> C
>>
>> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I'm not sure which of the instances went down, but it could be a few weeks.
>>>
>>> -JZ
>>>
>>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>>
>>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>>> was
>>>> the last time you restarted these instances?
>>>>
>>>> C
>>>>
>>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> It's not really out of memory. It's "unable to create new native
>>> thread".
>>>>> It appears to have to do with JMX (notice the RMI).
>>>>>
>>>>> -JZ
>>>>>
>>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>>
>>>>>> Not known specifically that I know of. Can you give us more details?
>>> Did
>>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>>> tell you?
>>>>>>
>>>>>> C
>>>>>>
>>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>>
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>>> Accept-0: accept loop for ServerSocket[addr>>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>>> 11:01:28
>>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start0(Native Method)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB