Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> java.lang.OutOfMemoryError: unable to create new native thread


Copy link to this message
-
Re: java.lang.OutOfMemoryError: unable to create new native thread
Can you provide the logfile and config?
On Tue, Nov 26, 2013 at 12:20 PM, Cochran, David <[EMAIL PROTECTED]>wrote:

> I've got a pretty good sized box collecting logs for a number of sources
> (about a dozen or so).
> Actually two instances were running on this box (one production and the
> other a testing environment)
>
> After adding a dozen more log files i'm running into this error (87 logs
> for PROD and 70 or so for Testing)
>
> Nov 26, 2013 10:25:27 AM
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink
> WARNING: Failed to accept a connection.
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
>  at java.lang.Thread.start(Thread.java:640)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
>  at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.start(AbstractNioWorker.java:160)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.register(AbstractNioWorker.java:131)
> at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.registerAcceptedChannel(NioServerSocketPipelineSink.java:269)
>  at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run(NioServerSocketPipelineSink.java:231)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> I have been able to confirm that they are not bumping into the OS ulimit,
> and increasing the sizes of the JVM does not make this go away
> (JAVA_OPTS="-Xms1g -Xmx6g -Dcom.sun.management.jmxremot").  I ended up
> having to shut down the testing environment instance so as not to lose any
> more production logs.
>
> The box is 16 CPU with 16G RAM using local disks, running RHEL5.x
>
> Anytime process threads exceed ~930 (the results of  ps -eLF -u flume | wc
> -l )the above error appears...if allowed to continue ultimately crashes the
> box.
>
> Any ideas on resolving this issue or have I run into a thread limit that
> cannot be gotten around without adding another server?
>
> Thanks,
> Dave
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB