Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # dev - Re: Review Request 14041: MetadataResponse during Consumer's Rebalance Process maybe Stale


+
Guozhang Wang 2013-09-09, 21:31
+
Neha Narkhede 2013-09-09, 23:44
+
Swapnil Ghike 2013-09-10, 06:53
+
Guozhang Wang 2013-09-10, 18:30
Copy link to this message
-
Re: Review Request 14041: MetadataResponse during Consumer's Rebalance Process maybe Stale
Joel Koshy 2013-09-10, 21:46

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14041/#review26022
-----------------------------------------------------------
Following up on the performance concerns that Neha had raised - this will be a significant bottleneck for tools such as the mirror-maker as the rebalance latency will almost certainly multiply.

We could consider utilizing the returned children information in ZkRebalancerListener's handleChildChange - i.e., curChilds will be the partitions if the parentPath is a topic. It won't be a trivial change then - we currently recompute the full map of topic/partitions during each rebalance. Now, we will need to "maintain" that data structure. There is also an initial bootstrap problem (that can also be addressed).

This makes me wonder if it is better to implement selective rebalance - that way the overhead of going to ZK for partition information (for the affected topics alone) should be acceptable. It will also give us the added benefit of reducing the overall time for rebalance on a topic event.
- Joel Koshy
On Sept. 10, 2013, 6:29 p.m., Guozhang Wang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14041/
> -----------------------------------------------------------
>
> (Updated Sept. 10, 2013, 6:29 p.m.)
>
>
> Review request for kafka.
>
>
> Bugs: KAFKA-1030
>     https://issues.apache.org/jira/browse/KAFKA-1030
>
>
> Repository: kafka
>
>
> Description
> -------
>
> Using the approach of reading directly from ZK.
>
>
> Diffs
> -----
>
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943
>
> Diff: https://reviews.apache.org/r/14041/diff/
>
>
> Testing
> -------
>
> unit tests
>
>
> Thanks,
>
> Guozhang Wang
>
>