Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Mirrormaker stopped consuming


Copy link to this message
-
Re: Mirrormaker stopped consuming
We have about 4 topics that are actively consumed each with 8 partitions. I
am able to see all topics/partitions and its leaders via list topics
command. But now the mirrormaker is restarted to restore the problem. I
will double check leaders for partitions when the problem happen again.
On Tue, Sep 10, 2013 at 6:12 PM, Sriram Subramanian <
[EMAIL PROTECTED]> wrote:

> Could you provide the topic/partition and also a dump of the zookeeper
> state? One possibility is if we never elected a leader for a topic
> partition.
>
> On 9/10/13 3:07 PM, "Rajasekar Elango" <[EMAIL PROTECTED]> wrote:
>
> >I did more investigation to confirm if this could be really due to jira
> >issue https://issues.apache.org/jira/browse/KAFKA-937 . But all thread
> >dumps I got did not had any dead locks for even blocked threads. All
> >threads are pretty much in RUNNABLE state. I consistently see
> >"kafka.common.NotLeaderForPartitionException"
> >in mirrormaker logs every time this happens. Could
> >NotLeaderForPartitionException
> >cause consumer threads to stop consuming..? What are the debugging steps
> >we
> >can try to identify root cause of this problem?
> >
> >Thanks,
> >Raja.
> >
> >
> >On Thu, Sep 5, 2013 at 7:33 PM, Rajasekar Elango
> ><[EMAIL PROTECTED]>wrote:
> >
> >> This jira seem to be resolved very recently. We are using snapshot of
> >>0.8
> >> release from beginning of june.
> >>
> >> Thanks,
> >> Raja.
> >>
> >>
> >> On Thu, Sep 5, 2013 at 1:49 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >>
> >>> Your threaddump looks similar to the deadlock in
> >>> https://issues.apache.org/jira/browse/KAFKA-937 I thought that fix was
> >>> included in the beta release - which version of Kafka are you using?
> >>>
> >>>
> >>> On Thu, Sep 5, 2013 at 8:35 AM, Rajasekar Elango
> >>><[EMAIL PROTECTED]>
> >>> wrote:
> >>> > Thanks , I am working on tuning GC options. This happened again today
> >>> and
> >>> > mirrormaker stopped consuming. This time the last exception was:
> >>> >
> >>> > 2013-09-05 07:01:52,931
> >>> >
> >>>
> >>>[ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378
> >>>308858270-7a2f6553-0-2]
> >>> > WARN  (kafka.consumer.ConsumerFetcherThread)  -
> >>> >
> >>>
> >>>[ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378
> >>>308858270-7a2f6553-0-2],
> >>> > error for partition [jmx,5] to broker
> >>> > 2kafka.common.NotLeaderForPartitionException
> >>> >         at
> >>> sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
> >>> >         at
> >>>
> >>>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
> >>>tructorAccessorImpl.java:27)
> >>> >         at
> >>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >>> >         at java.lang.Class.newInstance0(Class.java:355)
> >>> >         at java.lang.Class.newInstance(Class.java:308)
> >>> >         at
> >>> kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonf
> >>>un$apply$5.apply(AbstractFetcherThread.scala:158)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonf
> >>>un$apply$5.apply(AbstractFetcherThread.scala:158)
> >>> >         at kafka.utils.Logging$class.warn(Logging.scala:88)
> >>> >         at
> >>> kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply
> >>>AbstractFetcherThread.scala:157)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchReuest$4.apply(
> >>>AbstractFetcherThread.scala:113)
> >>> >         at
> >>> scala.collectionimmutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> >>> >         at
> >>>
> >>>scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347
> >>>)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherTh
Thanks,
Raja.