Kafka, mail # user - How kafka assign partition to stream - 2014-03-10, 17:23
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
How kafka assign partition to stream
In my environment,  I have 2 brokers and only 1 topic,  each brokers has 10
so there are 20 partitions in total.
I have 4 consumers in one consumer group,  each consumer use
createMessageStreams to create 10 streams, 40 streams in total.

Since partition can not be split,  so there are 20 streams are empty

We found 2 machines have 10 valid streams, but the other 2 have no valid
Is it expected?  Maybe each consumer has 5 valid streams is better for load

And I want to know if  one of the machines which have 10 valid streams is
dead,  will the the partitions be balanced to other machine dynamically?
 i.e. some empty stream in other machines will get data now.

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB