Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Clustered Kafka/Zookeeper setup not rebalancing all partitions


Copy link to this message
-
Re: Clustered Kafka/Zookeeper setup not rebalancing all partitions
Jun Rao 2013-04-24, 01:02
Could you run the tool kafka.tools.ConsumerOffsetChecker? It will show you
which consumer instance owns which partition.

Thanks,

Jun
On Tue, Apr 23, 2013 at 12:12 AM, Karl Kirch <[EMAIL PROTECTED]> wrote:

> Our setup is a three node cluster, we have a Kafka 0.7.2 (with 10
> partitions per node) and Zookeeper 3.3.4 on each node. When we first start
> up the cluster, there are no issues and things work just fine, but after a
> while things start acting a bit haywire. It looks like we start to miss
> messages that should be coming through the system. However on client
> restart, logging shows that only the partitions on broker 2 and 3 are being
> re-balanced (for the particular topic I'm connecting to). I'm confidant
> that there is nothing else connected to the cluster (as we've seen this a
> couple of times now). To make it more interesting, if I switch topics, the
> client will rebalance the partitions on all three brokers.
>
> Anyone know of any reason why this could occur?
>
> Thanks,
> Karl