Search Hadoop and all its sub project:

Switch to Threaded View
Subject: Partitions per topic per broker?
Hi guys, apologies in advance for the newb question:

I am running a 3 broker setup, and I have a topic configured with 100 partitions in the broker config. But I've noticed that what seems to happen is that each broker gets 100 partitions and it looks kind of like this in the consumer logs: 1-0, 1-1, 1-2, ... 1-99, 2-0, 2-1... etc. So the topic actually has 300 partitions, 100 per broker? Why? How does this work with the random partitioning scheme? I don't really understand the design behind this and my reading of the documentation hasn't seemed to turn up anything (though it's entirely possible I missed it).

Thanks in advance!

Ian Friedman
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB