Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Changing the number of partitions after a topic is created


+
Calvin Lei 2013-07-04, 15:12
+
Jun Rao 2013-07-05, 04:06
+
Timothy Chen 2013-07-08, 17:33
+
Jun Rao 2013-07-09, 03:58
+
Philip OToole 2013-07-04, 15:50
+
Calvin Lei 2013-07-04, 16:06
Copy link to this message
-
Re: Changing the number of partitions after a topic is created
Philip O'Toole 2013-07-04, 18:00
On Thu, Jul 4, 2013 at 12:06 PM, Calvin Lei <[EMAIL PROTECTED]> wrote:
> Thanks Phillip. I used the kafka-topic.sh to create the topic to have 1 partition. Would the changing the server properties still work in this case?

I'm not familiar with that script, but server.properties is what
matters. Change that and it will work.

>
> Also this setting sounds like a global overwrite, correct?

Yes, but it can be set on a topic-by-topic basis if you prefer. Read
the comments contained within the server.properties for guidance.

https://github.com/apache/kafka/blob/0.7/config/server.properties

>
> regards,
> Cal
>
>
> On Jul 4, 2013, at 11:49 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
>> If you can pause your Producers, simply change the partition count to
>> 10 in the Kafka server properties file, and restart it. If you use the
>> high-level consumer, it will automatically start draining all
>> partitions that exist for a given topic.
>>
>> And our Kafka producers always write to partition -1 for a given
>> topic, meaning they never care about the partition count, and need no
>> changes in circumstances like this.
>>
>> Philip
>>
>> On Thu, Jul 4, 2013 at 11:11 AM, Calvin Lei <[EMAIL PROTECTED]> wrote:
>>> Hi I have a few topics created with 1 partition. After running the cluster
>>> for a few days, I want to increase the partition to 10 to improve the
>>> consumer throughput. I learnt that it is not supported in 0.8.
>>>
>>> What is the recommendation of improving consumer throughput after a topic
>>> is created and the data volume increased?
>>>
>>> Regards,
>>> Cal
>

 
+
Jun Rao 2013-07-05, 19:08
+
Rajasekar Elango 2013-08-28, 15:47