We have found that, for our application, having a number of total partitions as a multiple of the number of consumer hosts is beneficial. Because of this, whenever we add or remove consumer hosts, we have to change the number of partitions in the server config.
What are best practices for changing the number of partitions? It seems like adding partitions is fine but removing partitions would result in data loss - am I right? Is that avoidable? Is it preferable to bring in new servers with new partitions? Anything else I should keep in mind on this issue? Thanks!
If you don't have a lot of topics, one thing you can do is to over-partition a topic.
Also, in 0.7, # of partitions grows with brokers. This is going to change in 0.8, in which # of partitions is specified at topic creation time and won't change as brokers change. One needs to use an admin DDL to change # of partitions.
On Mon, Jan 7, 2013 at 10:23 PM, David Ross <[EMAIL PROTECTED]> wrote:
Reducing # partitions is going to be tricky. The data for those dropped partitions will just be lost.
On Tue, Jan 8, 2013 at 1:24 PM, David Ross <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext