Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop Jobtracker job and UI hangs - Deadlock detection


Copy link to this message
-
Hadoop Jobtracker job and UI hangs - Deadlock detection
Hi,

JT memory reaches 6.68/8.89 GB and not able to submit the job and UI is not
loading at all. But didn't see any JT OOM exceptions.

Have taken the thread dump of Jobtracker, and the JT thread dump as follows:

Deadlock Detection:

Can't print deadlocks:null
Thread 25817: (state = BLOCKED)

   - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may
   be imprecise)
   - org.apache.hadoop.hdfs.LeaseRenewer.run(int) @bci=274, line=397
   (Compiled frame)
   - org.apache.hadoop.hdfs.LeaseRenewer.access$600(org.apache.hadoop.hdfs.LeaseRenewer,
   int) @bci=2, line=69 (Interpreted frame)
   - org.apache.hadoop.hdfs.LeaseRenewer$1.run() @bci=8, line=273
   (Interpreted frame)
   - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)

Locked ownable synchronizers:

   - None

Thread 25815: (state = BLOCKED)

   - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
   imprecise)
   - org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run()
   @bci=245, line=3000 (Compiled frame)

Locked ownable synchronizers:

   - None

Thread 25813: (state = BLOCKED)

   - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
   imprecise)
   - org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
   line=747 (Compiled frame)
   - org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
   (Compiled frame)

Locked ownable synchronizers:

   - None

Thread 25812: (state = BLOCKED)

   - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
   imprecise)
   - org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
   line=747 (Compiled frame)
   - org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
   (Compiled frame)

Locked ownable synchronizers:

   - None

Thread 25790: (state = BLOCKED)

   - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
   imprecise)

Thread 25788: (state = BLOCKED)

   - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
   imprecise)
   - org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
   line=747 (Compiled frame)
   - org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
   (Compiled frame)

Locked ownable synchronizers:

   - None

Thread 25786: (state = BLOCKED)

   - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be
   imprecise)
   - org.apache.hadoop.ipc.Client$Connection.waitForWork() @bci=59,
   line=747 (Compiled frame)
   - org.apache.hadoop.ipc.Client$Connection.run() @bci=55, line=789
   (Compiled frame)

Locked ownable synchronizers:

   - None

Thread 25761: (state = BLOCKED)

   - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0
   (Compiled frame; information may be imprecise)
   - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled
   frame)
   - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled
   frame)
   - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69
   (Compiled frame)
   - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Interpreted
   frame)
   - org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(java.nio.channels.SelectableChannel,
   int, long) @bci=46, line=332 (Interpreted frame)
   - org.apache.hadoop.net.SocketIOWithTimeout.doIO(java.nio.ByteBuffer,
   int) @bci=80, line=157 (Compiled frame)
   - org.apache.hadoop.net.SocketInputStream.read(java.nio.ByteBuffer)
   @bci=6, line=155 (Compiled frame)
   - org.apache.hadoop.net.SocketInputStream.read(byte[], int, int) @bci=7,
   line=128 (Compiled frame)
   - java.io.FilterInputStream.read(byte[], int, int) @bci=7, line=116
   (Interpreted frame)
   - org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(byte[],
   int, int) @bci=4, line=364 (Interpreted frame)
   - java.io.BufferedInputStream.fill() @bci=175, line=218 (Compiled frame)
   - java.io.BufferedInputStream.read() @bci=12, line=237 (Compiled frame)
   - java.io.DataInputStream.readInt() @bci=4, line=370 (Compiled frame)
   - org.apache.hadoop.ipc.Client$Connection.receiveResponse() @bci=19,
   line=845 (Compiled frame)
   - org.apache.hadoop.ipc.Client$Connection.run() @bci=62, line=790
   (Compiled frame)
And Jobtracker heap summary as follows:

using thread-local object allocation.
Parallel GC with 10 thread(s)

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 10737418240 (10240.0MB)
NewSize = 1310720 (1.25MB)
MaxNewSize = 17592186044415 MB
OldSize = 5439488 (5.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)

Heap Usage:
PS Young Generation
Eden Space:
capacity = 6488064 (6.1875MB)
used = 6488064 (6.1875MB)
free = 0 (0.0MB)
100.0% used
capacity = 9764864 (9.3125MB)
used = 0 (0.0MB)
free = 9764864 (9.3125MB)
0.0% used
To Space:
capacity = 9764864 (9.3125MB)
used = 0 (0.0MB)
free = 9764864 (9.3125MB)
0.0% used
PS Old Generation
capacity = 7158300672 (6826.6875MB)
used = 7158240200 (6826.629829406738MB)
free = 60472 (0.05767059326171875MB)
99.99915521849708% used
PS Perm Generation
capacity = 26738688 (25.5MB)
used = 26428648 (25.204322814941406MB)
free = 310040 (0.29567718505859375MB)
98.8404816272212% used

Please help. It affects our production system.
Regards,
Viswa.J