Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Child processes on datanodes/task trackers


+
Sedighe Tabatabaei 2012-12-22, 10:16
Copy link to this message
-
Re: Child processes on datanodes/task trackers
This is an interesting stacktrace. The JVM had called exit but there
are threads stuck. Unsure if your JVM version (6u22) has anything to
do with it.

Are you using the JVM reuse feature?

On Sat, Dec 22, 2012 at 3:46 PM, Sedighe Tabatabaei
<[EMAIL PROTECTED]> wrote:
> Hello
> I wanted to know why I see running Child processes on my datanodes
> even though there is no job running at that time.
> when i use  jstack  <pid> command,result is:
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):
>
> "Attach Listener" daemon prio=10 tid=0x00007f2a74002000 nid=0x1f3
> waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "SIGTERM handler" daemon prio=10 tid=0x00007f2a74001000 nid=0x2fae
> waiting for monitor entry [0x00007f2a79891000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at java.lang.Shutdown.exit(Shutdown.java:168)
>         - waiting to lock <0x00007f2a9ffa68d0> (a java.lang.Class for
> java.lang.Shutdown)
>         at java.lang.Terminator$1.handle(Terminator.java:35)
>         at sun.misc.Signal$1.run(Signal.java:195)
>         at java.lang.Thread.run(Thread.java:662)
>
> "Thread-16" prio=10 tid=0x00007f2ae83d1000 nid=0x2f85 runnable
> [0x00007f2a9cb5a000]
>    java.lang.Thread.State: RUNNABLE
>         at java.lang.Throwable.fillInStackTrace(Native Method)
>         - locked <0x00007f2ad36e4d50> (a java.util.ConcurrentModificationException)
>         at java.lang.Throwable.<init>(Throwable.java:181)
>         at java.lang.Exception.<init>(Exception.java:29)
>         at java.lang.RuntimeException.<init>(RuntimeException.java:32)
>         at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
>         at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
>         at java.util.HashMap$KeyIterator.next(HashMap.java:828)
>         at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:315)
>
> "DestroyJavaVM" prio=10 tid=0x00007f2ae8009800 nid=0x1113 in
> Object.wait() [0x00007f2aedb81000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa5248528> (a
> com.sun.media.jai.codec.TempFileCleanupThread)
>         at java.lang.Thread.join(Thread.java:1186)
>         - locked <0x00007f2aa5248528> (a com.sun.media.jai.codec.TempFileCleanupThread)
>         at java.lang.Thread.join(Thread.java:1239)
>         at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
>         at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
>         at java.lang.Shutdown.runHooks(Shutdown.java:79)
>         at java.lang.Shutdown.sequence(Shutdown.java:123)
>         at java.lang.Shutdown.shutdown(Shutdown.java:190)
>         - locked <0x00007f2a9ffa68d0> (a java.lang.Class for java.lang.Shutdown)
>
> "Java2D Disposer" daemon prio=10 tid=0x00007f2ae8415000 nid=0x1775 in
> Object.wait() [0x00007f2a7a16d000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0x00007f2aa528a748> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>         at sun.java2d.Disposer.run(Disposer.java:127)
>         at java.lang.Thread.run(Thread.java:662)
>
> "Thread for syncLogs" daemon prio=10 tid=0x00007f2ae83bf000 nid=0x1208
> waiting on condition [0x00007f2a9f2fa000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.mapred.Child$2.run(Child.java:87)
>
> "Low Memory Detector" daemon prio=10 tid=0x00007f2ae8093000 nid=0x112f
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00007f2ae8090800 nid=0x112d

Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB