1) Should a producer be aware of which broker to write to or is this somehow managed by Kafka itself. For example, If I have 2 brokers with a configured partition size of 1 will my messages be written in a round-robin type of fashion to each broker or is this completed dependent on which broker the consume connects to?
2) Similar question goes for a writing to a topic that has multiple partitions. Are the writes randomly load-balanced between partitions or is this something the client needs to do?
3) If I have 1 topic with 1 partition and 2 brokers how does consuming messages work. For example, if my consumer connects to broker 1 how does it consume messages written to broker 2?
On Thu, Aug 29, 2013 at 11:09 AM, Mark <[EMAIL PROTECTED]> wrote: Sounds like a question regarding the Producer, which I am not familiar with. We use our own Producers here, written in C++.
It depends. Producers can specify the partition, or use a partition of '-1', which instructs Kafka to randomly choose a partition.
Don't think of partitions on one broker and partitions on another broker. Simply think of the aggregate partition count across the cluster. The high-level consumer that comes with Kafka will learn where all the partitions are for a given topic via Zookeeper, create connections to the Kafka brokers as needed, and drain all the partitions.
On Thu, Aug 29, 2013 at 1:28 PM, Mark <[EMAIL PROTECTED]> wrote: Plain ol' ConsumerConnector does it for us, in our Java-based high-level consumers.
Philip OToole 2013-08-29, 20:41
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext