Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper, mail # user - OOM/no more threads?


+
Jordan Zimmerman 2012-07-09, 02:29
+
Camille Fournier 2012-07-09, 02:43
+
Jordan Zimmerman 2012-07-09, 03:04
+
Camille Fournier 2012-07-09, 03:06
+
Jordan Zimmerman 2012-07-09, 03:08
+
Camille Fournier 2012-07-09, 03:11
+
Jordan Zimmerman 2012-07-09, 03:13
+
Rakesh R 2012-07-09, 05:07
Copy link to this message
-
Re: OOM/no more threads?
Jordan Zimmerman 2012-07-09, 07:27
-Xms35004M -Xmx35004M -Xss128k

64 bit Centos

On Jul 8, 2012, at 10:07 PM, Rakesh R wrote:

> Hi Jordan,
>
> BTW, could you give details of -Xmx, -Xss  parameters?  OS ? 32bit/62bit system?
>
> As I understand the default value of thread stack -Xss depends on OS/JVM. As number of threads increase, memory usage increases and also can lead to 'out ofMemoryError'.
>
> Also, there could be chances of thread leak in the system.
>
> -Rakesh
> ________________________________________
> From: Jordan Zimmerman [[EMAIL PROTECTED]]
> Sent: Monday, July 09, 2012 8:43 AM
> To: [EMAIL PROTECTED]
> Subject: Re: OOM/no more threads?
>
> Good idea - I'll try that.
>
> -JZ
>
> On Jul 8, 2012, at 8:11 PM, Camille Fournier wrote:
>
>> What does a jstack on another long-running instance show? Anything out of
>> the norm?
>>
>> C
>>
>> On Sun, Jul 8, 2012 at 11:08 PM, Jordan Zimmerman <
>> [EMAIL PROTECTED]> wrote:
>>
>>> I'm not sure which of the instances went down, but it could be a few weeks.
>>>
>>> -JZ
>>>
>>> On Jul 8, 2012, at 8:06 PM, Camille Fournier wrote:
>>>
>>>> Yeah a tickle in the back of my mind says that this is a JVM bug. When
>>> was
>>>> the last time you restarted these instances?
>>>>
>>>> C
>>>>
>>>> On Sun, Jul 8, 2012 at 11:04 PM, Jordan Zimmerman <
>>>> [EMAIL PROTECTED]> wrote:
>>>>
>>>>> It's not really out of memory. It's "unable to create new native
>>> thread".
>>>>> It appears to have to do with JMX (notice the RMI).
>>>>>
>>>>> -JZ
>>>>>
>>>>> On Jul 8, 2012, at 7:43 PM, Camille Fournier wrote:
>>>>>
>>>>>> Not known specifically that I know of. Can you give us more details?
>>> Did
>>>>>> you actually run out of memory? File descriptors? What does a heap dump
>>>>>> tell you?
>>>>>>
>>>>>> C
>>>>>>
>>>>>> On Sun, Jul 8, 2012 at 10:29 PM, Jordan Zimmerman <
>>>>>> [EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>>> Is this a known issue in 3.3.3? If so, is it fixed in 3.3.5?
>>>>>>>
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: WARNING: RMI TCP
>>>>>>> Accept-0: accept loop for ServerSocket[addr>>>>>>> 0.0.0.0/0.0.0.0,port=0,localport=59462] throws
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: Jul 8, 2012
>>>>> 11:01:28
>>>>>>> PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.run(Thread.java:636)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:402)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>>
>>>>>
>>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start(Thread.java:614)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server: at
>>>>>>> java.lang.Thread.start0(Native Method)
>>>>>>> Sun Jul 08 23:01:28 GMT 2012 ERROR ZooKeeper Server:
>>>>>>> java.lang.OutOfMemoryError: unable to create new native thread
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
+
Rakesh R 2012-07-09, 13:36