We are trying to add a broker to a 10 node cluster. We have 7 different topics, each of them is divided in 10 partitions, and their replication factor is 3.
To send traffic to this new node, we tried the kafka-reassign-partitions.sh tool, but for some reason, it doesnt work, and now it seems that we are stuck in some kind of partition reassignment process. We can't see any data on our new node, and when we try to execute the partition reassignment tool for another topic it says :
Partitions reassignment failed due to Partition reassignment currently in progress for Map([AD_EVENTS,5] -> ReassignedPartitionsContext(List(2),null), [AD_EVENTS,3] -> ReassignedPartitionsContext(List(1),null), [AD_EVENTS,4] -> ReassignedPartitionsContext(List(3),null)). Aborting operation kafka.common.AdminCommandFailedException: Partition reassignment currently in progress for Map([AD_EVENTS,5] -> ReassignedPartitionsContext(List(2),null), [AD_EVENTS,3] -> ReassignedPartitionsContext(List(1),null), [AD_EVENTS,4] -> ReassignedPartitionsContext(List(3),null)). Aborting operation at kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:201) at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:140) at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala) Any help would be greatly appreciated !
We used the code checked in this branch a few hours before the official 0.8.0 final release : https://github.com/apache/kafka/tree/0.8 So hopefully it should be the exact same code as the official release.
The controller logs are empty.
In a previous exchange you advised us to not use trunk unless we felt adventurous. So if you think 0.8.1 is production ready, and that it shouldn't crash unexpectedly, we can probably give it a try. Otherwise, since our data pipeline heavily relies on Kafka, we might have to use another solution... (bring up a brand new cluster each time we want to resize the cluster ?)
I wouldn't call 0.8.1 production ready just yet. We are still in the process of deploying it at LinkedIn. Until it is ready, there isn't a good cluster expansion solution other than spinning up a new cluster. This is probably a little easier if you have a VIP in front of your kafka cluster.
Thanks, Neha On Mon, Dec 9, 2013 at 9:40 AM, Maxime Nay <[EMAIL PROTECTED]> wrote:
That's good to know. Thanks for your help! 2013/12/9 Neha Narkhede <[EMAIL PROTECTED]>
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext