Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> [jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers


+
Jun Rao 2013-01-16, 17:48
+
Jun Rao 2013-01-13, 22:34
Copy link to this message
-
[jira] [Commented] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

    [ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13554068#comment-13554068 ]

Jun Rao commented on KAFKA-693:
-------------------------------

Thanks for the patch. Some comments:

10. ZookeeperConsumerConnector: Let's define a constant InvalidOffset, instead of using -1 directly.

11. ConsumerFetcherManager.doWork(): After we identify the leader of a partition, the leader could change immediately. So, we may hit the exception when calling addFetcher(). When this happens, we haven't added the partition to the fetcher and we don't want to lose it. So, we should add it back to noLeaderPartitionSet so that we can find the new leader later.

12. ReplicaFetcherThread: Yes, it should also throw an exception if getOffsetBefore returns an error.

13. AbstractFetcherThread.doWork(): We need to handle the exception when calling handleOffsetOutOfRange(). If we get an exception, we should add the partition to partitionsWithError. This will cover both ConsumerFetcherThread and ReplicaFetcherThread.

                
> Consumer rebalance fails if no leader available for a partition and stops all fetchers
> --------------------------------------------------------------------------------------
>
>                 Key: KAFKA-693
>                 URL: https://issues.apache.org/jira/browse/KAFKA-693
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.8
>            Reporter: Maxime Brugidou
>            Assignee: Maxime Brugidou
>         Attachments: KAFKA-693.patch, mirror_debug.log, mirror.log
>
>
> I am currently experiencing this with the MirrorMaker but I assume it happens for any rebalance. The symptoms are:
> I have replication factor of 1
> 1. If i start the MirrorMaker (bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config mirror-consumer.properties  --producer.config mirror-producer.properties --blacklist 'xdummyx' --num.streams=1 --num.producers=1) with a broker down
> 1.1 I set the refresh.leader.backoff.ms to 600000 (10min) so that the ConsumerFetcherManager doesn't retry to often to get the unavailable partitions
> 1.2 The rebalance starts at the init step and fails: Exception in thread "main" kafka.common.ConsumerRebalanceFailedException: KafkaMirror_mirror-01-1357893495345-fac86b15 can't rebalance after 4 retries
> 1.3 After the exception, everything stops (fetchers and queues)
> 1.4 I attached the full logs (info & debug) for this case
> 2. If i start the MirrorMaker with all the brokers up and then kill a broker
> 2.1 The first rebalance is successful
> 2.2 The consumer will handle correctly the broker down and stop the associated ConsumerFetcherThread
> 2.3 The refresh.leader.backoff.ms to 600000 works correctly
> 2.4 If something triggers a rebalance (new topic, partition reassignment...), then we go back to 1., the rebalance fails and stops everything.
> I think the desired behavior is to consumer whatever is available, and try later at some intervals. I would be glad to help on that issue although the Consumer code seems a little tough to get on.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB