Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How kafka assign partition to stream


Copy link to this message
-
How kafka assign partition to stream
In my environment,  I have 2 brokers and only 1 topic,  each brokers has 10
partitions,
so there are 20 partitions in total.
I have 4 consumers in one consumer group,  each consumer use
createMessageStreams to create 10 streams, 40 streams in total.

Since partition can not be split,  so there are 20 streams are empty
streams.

We found 2 machines have 10 valid streams, but the other 2 have no valid
streams.
Is it expected?  Maybe each consumer has 5 valid streams is better for load
balance.

And I want to know if  one of the machines which have 10 valid streams is
dead,  will the the partitions be balanced to other machine dynamically?
 i.e. some empty stream in other machines will get data now.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB