Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Mirrormaker stopped consuming


Copy link to this message
-
Re: Mirrormaker stopped consuming
This jira seem to be resolved very recently. We are using snapshot of 0.8
release from beginning of june.

Thanks,
Raja.
On Thu, Sep 5, 2013 at 1:49 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> Your threaddump looks similar to the deadlock in
> https://issues.apache.org/jira/browse/KAFKA-937 I thought that fix was
> included in the beta release - which version of Kafka are you using?
>
>
> On Thu, Sep 5, 2013 at 8:35 AM, Rajasekar Elango <[EMAIL PROTECTED]>
> wrote:
> > Thanks , I am working on tuning GC options. This happened again today and
> > mirrormaker stopped consuming. This time the last exception was:
> >
> > 2013-09-05 07:01:52,931
> >
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378308858270-7a2f6553-0-2]
> > WARN  (kafka.consumer.ConsumerFetcherThread)  -
> >
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378308858270-7a2f6553-0-2],
> > error for partition [jmx,5] to broker
> > 2kafka.common.NotLeaderForPartitionException
> >         at
> sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
> >         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >         at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >         at java.lang.Class.newInstance0(Class.java:355)
> >         at java.lang.Class.newInstance(Class.java:308)
> >         at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> >         at kafka.utils.Logging$class.warn(Logging.scala:88)
> >         at
> kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> >         at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> >         at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> >         at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> >         at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> >         at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >
> >
> >
> > I also captured thread dump and pasting it below. Can you confirm if
> > this could again due to Full GC or something else that need to be
> > looked at.
> >
> >
> >
> >
> > ull thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
> >
> > "Attach Listener" daemon prio=10 tid=0x00007f539c001800 nid=0x6b49
> > waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> >
> "ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378308858270-7a2f6553-0-3"
> > prio=10 tid=0x00007f52f800c000 nid=0x5e03 waiting on condition
> > [0x00007f53b46cf000]
> >    java.lang.Thread.State: WAITING (parking)
> >         at sun.misc.Unsafe.park(Native Method)
> >         - parking to wait for  <0x00007f541d14f5a0> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> >         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> >         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
> >         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
> >         at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)

Thanks,
Raja.