Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Recommendation for number of brokers on kafka(0.7.2) hosts


Copy link to this message
-
Re: Recommendation for number of brokers on kafka(0.7.2) hosts
1) Will setting 4 brokers per host with different ports and different log
data directories be beneficial to use all the available space?
2) Will there be any disadvantage using multiple brokers on same host?

It is recommended that you do not deploy multiple brokers on the same box
since that will not lead to optimal page cache usage, which will affect IO
performance.

3) If partition count is set to 2 per topic. Having multiple brokers on
each machine , multiply the partition count per machine? i.e with 4 brokers
per machine on 5 machines will generate 2*4*5 = 40 partitions per topic?

Correct.

4) Is there a way to use all the directories from a single broker?

Kafka 0.8 supports specifying multiple log directories through the log.dirs
config.

Thanks,
Neha
On Tue, Oct 1, 2013 at 1:17 PM, rk vishu <[EMAIL PROTECTED]> wrote:

> Hello All,
>
> I am currently using 5 node kafka cluster with 0.7.2 version. Would like to
> get some advice on optimal number of brokers on each kafka host. Below is
> the specification of each machine
>
> - 4 data directories /data1,/data2, /data3, /data4 with 200+GB usable
> space. RAID10
> - 24 Core CPU
> - 32GB RAM
>
> From the server properties, i believe we can only configure one data
> log.dir.
> So below are my questions
> 1) Will setting 4 brokers per host with different ports and different log
> data directories be beneficial to use all the available space?
> 2) Will there be any disadvantage using multiple brokers on same host?
> 3) If partition count is set to 2 per topic. Having multiple brokers on
> each machine , multiply the partition count per machine? i.e with 4 brokers
> per machine on 5 machines will generate 2*4*5 = 40 partitions per topic?
> 4) Is there a way to use all the directories from a single broker?
>
> Thanks and Regards
> Ravi
>