Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Mirrormaker stopped consuming


Copy link to this message
-
Re: Mirrormaker stopped consuming
This jira seem to be resolved very recently. We are using snapshot of 0.8
release from beginning of june.

Thanks,
Raja.
On Thu, Sep 5, 2013 at 1:49 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> Your threaddump looks similar to the deadlock in
> https://issues.apache.org/jira/browse/KAFKA-937 I thought that fix was
> included in the beta release - which version of Kafka are you using?
>
>
> On Thu, Sep 5, 2013 at 8:35 AM, Rajasekar Elango <[EMAIL PROTECTED]>
> wrote:
> > Thanks , I am working on tuning GC options. This happened again today and
> > mirrormaker stopped consuming. This time the last exception was:
> >
> > 2013-09-05 07:01:52,931
> >
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378308858270-7a2f6553-0-2]
> > WARN  (kafka.consumer.ConsumerFetcherThread)  -
> >
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378308858270-7a2f6553-0-2],
> > error for partition [jmx,5] to broker
> > 2kafka.common.NotLeaderForPartitionException
> >         at
> sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
> >         at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >         at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >         at java.lang.Class.newInstance0(Class.java:355)
> >         at java.lang.Class.newInstance(Class.java:308)
> >         at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonfun$apply$5.apply(AbstractFetcherThread.scala:158)
> >         at kafka.utils.Logging$class.warn(Logging.scala:88)
> >         at
> kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:157)
> >         at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply(AbstractFetcherThread.scala:113)
> >         at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> >         at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347)
> >         at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:113)
> >         at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:89)
> >         at
> kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51)
> >
> >
> >
> > I also captured thread dump and pasting it below. Can you confirm if
> > this could again due to Full GC or something else that need to be
> > looked at.
> >
> >
> >
> >
> > ull thread dump Java HotSpot(TM) 64-Bit Server VM (17.0-b16 mixed mode):
> >
> > "Attach Listener" daemon prio=10 tid=0x00007f539c001800 nid=0x6b49
> > waiting on condition [0x0000000000000000]
> >    java.lang.Thread.State: RUNNABLE
> >
> >
> "ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378308858270-7a2f6553-0-3"
> > prio=10 tid=0x00007f52f800c000 nid=0x5e03 waiting on condition
> > [0x00007f53b46cf000]
> >    java.lang.Thread.State: WAITING (parking)
> >         at sun.misc.Unsafe.park(Native Method)
> >         - parking to wait for  <0x00007f541d14f5a0> (a
> > java.util.concurrent.locks.ReentrantLock$NonfairSync)
> >         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> >         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> >         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
> >         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
> >         at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)

Thanks,
Raja.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB