Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Chukwa >> mail # user >> Deadlock happening in ChukwaDailyRollingFileAppender


Copy link to this message
-
Re: Deadlock happening in ChukwaDailyRollingFileAppender
Does SocketAppender ecsape newlines? Like, log an exception with a stack
trace(multiple lines) as a single log statement? And can it register an
adapter automatically?
On Jun 24, 2012 12:20 PM, "Eric Yang" <[EMAIL PROTECTED]> wrote:

> Hi Abhijit,
>
> The dead lock happens when there are too many registrations with the
> agent and agent has exceeded the open file descriptor limit.  I would
> recommend to use Log4J SocketAppender with combination of
> SocketAdaptor.  It reduces the overhead of file offset tracking and
> reduces the number of file descriptors used by the agent.  It works
> better than using ChukwaDailyFileAppender and file adaptor
> combination.
>
> regards,
> Eric
>
> On Fri, Jun 22, 2012 at 9:50 AM, Abhijit Dhar <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > I am getting a deadlock in ChukwaDailyRollingFileAppender. The thread
> dump
> > is:
> >
> > Found one Java-level deadlock:
> > ============================> > "Thread-4":
> >  waiting to lock monitor 0x00007f1970004818 (object 0x00000000cb6669e8, a
> > java.lang.Class),
> >  which is held by "Thread for syncLogs"
> > "Thread for syncLogs":
> >  waiting to lock monitor 0x00007f1970004ea8 (object 0x00000000d65c44b8, a
> > org.apache.log4j.Logger),
> >  which is held by "main"
> > "main":
> >  waiting to lock monitor 0x00007f1970003d98 (object 0x00000000d65c3e80, a
> > org.apache.log4j.Logger),
> >  which is held by "communication thread"
> > "communication thread":
> >  waiting to lock monitor 0x0000000041eba250 (object 0x00000000d6736ec0, a
> > java.lang.Object),
> >  which is held by "main"
> >
> > Java stack information for the threads listed above:
> > ==================================================> > "Thread-4":
> >        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:213)
> >        - waiting to lock <0x00000000cb6669e8> (a java.lang.Class for
> > org.apache.hadoop.mapred.TaskLog)
> >        at org.apache.hadoop.mapred.Child$1.run(Child.java:75)
> > "Thread for syncLogs":
> >        at org.apache.log4j.Category.getAllAppenders(Category.java:409)
> >        - waiting to lock <0x00000000d65c44b8> (a org.apache.log4j.Logger)
> >        at org.apache.hadoop.mapred.TaskLog.syncLogs(TaskLog.java:218)
> >        - locked <0x00000000cb6669e8> (a java.lang.Class for
> > org.apache.hadoop.mapred.TaskLog)
> >        at org.apache.hadoop.mapred.Child$2.run(Child.java:89)
> > "main":
> >        at org.apache.log4j.Category.callAppenders(Category.java:202)
> >        - waiting to lock <0x00000000d65c3e80> (a org.apache.log4j.Logger)
> >        at org.apache.log4j.Category.forcedLog(Category.java:388)
> >        at org.apache.log4j.Category.warn(Category.java:1008)
> >        at
> >
> org.apache.hadoop.chukwa.datacollection.controller.ChukwaAgentController.addByName(ChukwaAgentController.java:267)
> >        at
> >
> org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender.subAppend(ChukwaDailyRollingFileAppender.java:565)
> >        - locked <0x00000000d6736ec0> (a java.lang.Object)
> >        at org.apache.log4j.WriterAppender.append(WriterAppender.java:159)
> >        at
> > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:230)
> >        - locked <0x00000000d65b38c0> (a
> > com.glassdoor.chukwa.inputtools.log4j.GDChukwaDailyRollingFileAppender)
> >        at
> >
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65)
> >        at org.apache.log4j.Category.callAppenders(Category.java:203)
> >        - locked <0x00000000d65c44b8> (a org.apache.log4j.Logger)
> >        at org.apache.log4j.Category.forcedLog(Category.java:388)
> >        at org.apache.log4j.Category.log(Category.java:853)
> >        at
> > org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:222)
> >        at
> >
> com.glassdoor.util.netsuite.service.NetSuiteServiceImpl.init(NetSuiteServiceImpl.java:86)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at