Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Chukwa, mail # user - Deadlock happening in ChukwaDailyRollingFileAppender


Copy link to this message
-
Deadlock happening in ChukwaDailyRollingFileAppender
Abhijit Dhar 2012-06-22, 16:50
Hi,

I am getting a deadlock in ChukwaDailyRollingFileAppender. The thread dump
is:

Found one Java-level deadlock:
============================"Thread-4":
  waiting to lock monitor 0x00007f1970004818 (object 0x00000000cb6669e8, a
java.lang.Class),
  which is held by "Thread for syncLogs"
"Thread for syncLogs":
  waiting to lock monitor 0x00007f1970004ea8 (object 0x00000000d65c44b8, a
org.apache.log4j.Logger),
  which is held by "main"
"main":
  waiting to lock monitor 0x00007f1970003d98 (object 0x00000000d65c3e80, a
org.apache.log4j.Logger),
  which is held by "communication thread"
"communication thread":
  waiting to lock monitor 0x0000000041eba250 (object 0x00000000d6736ec0, a
java.lang.Object),
  which is held by "main"

Java stack information for the threads listed above:
=================================================="Thread-4":
        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:213)
        - waiting to lock <0x00000000cb6669e8> (a java.lang.Class for
org.apache.hadoop.mapred.TaskLog)
        at org.apache.hadoop.mapred.Child$1.run(Child.java:75)
"Thread for syncLogs":
        at org.apache.log4j.Category.getAllAppenders(Category.java:409)
        - waiting to lock <0x00000000d65c44b8> (a org.apache.log4j.Logger)
        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:218)
        - locked <0x00000000cb6669e8> (a java.lang.Class for
org.apache.hadoop.mapred.TaskLog)
        at org.apache.hadoop.mapred.Child$2.run(Child.java:89)
"main":
        at org.apache.log4j.Category.callAppenders(Category.java:202)
        - waiting to lock <0x00000000d65c3e80> (a org.apache.log4j.Logger)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.warn(Category.java:1008)
        at
org.apache.hadoop.chukwa.datacollection.controller.ChukwaAgentController.addByName(ChukwaAgentController.java:267)
        at
org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender.subAppend(ChukwaDailyRollingFileAppender.java:565)
        - locked <0x00000000d6736ec0> (a java.lang.Object)
        at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
        at
org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
        - locked <0x00000000d65b38c0> (a
com.glassdoor.chukwa.inputtools.log4j.GDChukwaDailyRollingFileAppender)
        at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
        at org.apache.log4j.Category.callAppenders(Category.java:203)
        - locked <0x00000000d65c44b8> (a org.apache.log4j.Logger)
        at org.apache.log4j.Category.forcedLog(Category.java:388)
        at org.apache.log4j.Category.log(Category.java:853)
        at
org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:222)
        at
com.glassdoor.util.netsuite.service.NetSuiteServiceImpl.init(NetSuiteServiceImpl.java:86)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1378)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1339)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1299)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:463)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:404)
        at java.security.AccessController.doPrivileged(Native Method)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:375)
        at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:263)
        at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:170)
        - locked <0x00000000f77044f0> (a
java.util.concurrent.ConcurrentHashMap)
        at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:260)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:184)
        at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:163)
        at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:430)
        - locked <0x00000000f7704520> (a
java.util.concurrent.ConcurrentHashMap)
        at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:729)
        at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:381)
        - locked <0x00000000f77045d8> (a java.lang.Object)
        at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
        at
org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
        at
com.glassdoor.util.hadoop.entity.GDHadoopConfiguration.getAppContext(GDHadoopConfiguration.java:74)
        - locked <0x00000000d6736ef0> (a java.lang.Object)
        at
com.glassdoor.util.hadoop.entity.SpringHadoopMapperWrapper.run(SpringHadoopMapperWrapper.java:116)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
+
Eric Yang 2012-06-24, 19:20
+
Abhijit Dhar 2012-06-25, 01:27
+
Eric Yang 2012-06-25, 02:55
+
Abhijit Dhar 2012-07-06, 01:01
+
Eric Yang 2012-07-06, 05:21