Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Mirrormaker stopped consuming


+
Rajasekar Elango 2013-09-03, 15:59
+
Neha Narkhede 2013-09-03, 16:26
+
Rajasekar Elango 2013-09-03, 17:40
+
Neha Narkhede 2013-09-04, 01:25
+
Rajasekar Elango 2013-09-05, 15:36
+
Joel Koshy 2013-09-05, 17:50
+
Rajasekar Elango 2013-09-05, 23:34
+
Rajasekar Elango 2013-09-10, 22:08
+
Joel Koshy 2013-09-10, 23:12
+
Joel Koshy 2013-09-10, 23:16
Copy link to this message
-
Re: Mirrormaker stopped consuming
We have about 4 topics that are actively consumed each with 8 partitions. I
am able to see all topics/partitions and its leaders via list topics
command. But now the mirrormaker is restarted to restore the problem. I
will double check leaders for partitions when the problem happen again.
On Tue, Sep 10, 2013 at 6:12 PM, Sriram Subramanian <
[EMAIL PROTECTED]> wrote:

> Could you provide the topic/partition and also a dump of the zookeeper
> state? One possibility is if we never elected a leader for a topic
> partition.
>
> On 9/10/13 3:07 PM, "Rajasekar Elango" <[EMAIL PROTECTED]> wrote:
>
> >I did more investigation to confirm if this could be really due to jira
> >issue https://issues.apache.org/jira/browse/KAFKA-937 . But all thread
> >dumps I got did not had any dead locks for even blocked threads. All
> >threads are pretty much in RUNNABLE state. I consistently see
> >"kafka.common.NotLeaderForPartitionException"
> >in mirrormaker logs every time this happens. Could
> >NotLeaderForPartitionException
> >cause consumer threads to stop consuming..? What are the debugging steps
> >we
> >can try to identify root cause of this problem?
> >
> >Thanks,
> >Raja.
> >
> >
> >On Thu, Sep 5, 2013 at 7:33 PM, Rajasekar Elango
> ><[EMAIL PROTECTED]>wrote:
> >
> >> This jira seem to be resolved very recently. We are using snapshot of
> >>0.8
> >> release from beginning of june.
> >>
> >> Thanks,
> >> Raja.
> >>
> >>
> >> On Thu, Sep 5, 2013 at 1:49 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> >>
> >>> Your threaddump looks similar to the deadlock in
> >>> https://issues.apache.org/jira/browse/KAFKA-937 I thought that fix was
> >>> included in the beta release - which version of Kafka are you using?
> >>>
> >>>
> >>> On Thu, Sep 5, 2013 at 8:35 AM, Rajasekar Elango
> >>><[EMAIL PROTECTED]>
> >>> wrote:
> >>> > Thanks , I am working on tuning GC options. This happened again today
> >>> and
> >>> > mirrormaker stopped consuming. This time the last exception was:
> >>> >
> >>> > 2013-09-05 07:01:52,931
> >>> >
> >>>
> >>>[ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378
> >>>308858270-7a2f6553-0-2]
> >>> > WARN  (kafka.consumer.ConsumerFetcherThread)  -
> >>> >
> >>>
> >>>[ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-2-asg.ops.sfdc.net-1378
> >>>308858270-7a2f6553-0-2],
> >>> > error for partition [jmx,5] to broker
> >>> > 2kafka.common.NotLeaderForPartitionException
> >>> >         at
> >>> sun.reflect.GeneratedConstructorAccessor18.newInstance(Unknown Source)
> >>> >         at
> >>>
> >>>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons
> >>>tructorAccessorImpl.java:27)
> >>> >         at
> >>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >>> >         at java.lang.Class.newInstance0(Class.java:355)
> >>> >         at java.lang.Class.newInstance(Class.java:308)
> >>> >         at
> >>> kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:70)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonf
> >>>un$apply$5.apply(AbstractFetcherThread.scala:158)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4$$anonf
> >>>un$apply$5.apply(AbstractFetcherThread.scala:158)
> >>> >         at kafka.utils.Logging$class.warn(Logging.scala:88)
> >>> >         at
> >>> kafka.utils.ShutdownableThread.warn(ShutdownableThread.scala:23)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$4.apply
> >>>AbstractFetcherThread.scala:157)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread$$anonfun$processFetchReuest$4.apply(
> >>>AbstractFetcherThread.scala:113)
> >>> >         at
> >>> scala.collectionimmutable.HashMap$HashMap1.foreach(HashMap.scala:178)
> >>> >         at
> >>>
> >>>scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:347
> >>>)
> >>> >         at
> >>>
> >>>kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherTh
Thanks,
Raja.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB