Hi I have a few topics created with 1 partition. After running the cluster for a few days, I want to increase the partition to 10 to improve the consumer throughput. I learnt that it is not supported in 0.8.
What is the recommendation of improving consumer throughput after a topic is created and the data volume increased?
If you can pause your Producers, simply change the partition count to 10 in the Kafka server properties file, and restart it. If you use the high-level consumer, it will automatically start draining all partitions that exist for a given topic.
And our Kafka producers always write to partition -1 for a given topic, meaning they never care about the partition count, and need no changes in circumstances like this.
On Thu, Jul 4, 2013 at 11:11 AM, Calvin Lei <[EMAIL PROTECTED]> wrote:
On Thu, Jul 4, 2013 at 12:06 PM, Calvin Lei <[EMAIL PROTECTED]> wrote:
I'm not familiar with that script, but server.properties is what matters. Change that and it will work. Yes, but it can be set on a topic-by-topic basis if you prefer. Read the comments contained within the server.properties for guidance.
Currently, once a topic is created, the number of partitions can't be changed. We are working on a tool to allow that. For now, you will have to either use a new topic with more partitions or if you don't care about messages being consumed in order, you can feed the consumed messages to a separate thread pool for processing.
Jun On Thu, Jul 4, 2013 at 8:11 AM, Calvin Lei <[EMAIL PROTECTED]> wrote:
We are shooting for providing a patch in a couple of weeks.
Jun On Mon, Jul 8, 2013 at 10:33 AM, Timothy Chen <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext