Now neither of these integrate with Zookeeper so we are missing quite a few features:
- Auto discovery of brokers - Auto discover of partitions for consumers - … fill in the rest here, new to Kafka so don't know everything that is missing
I was wondering what my best options are going forward with Kakfa? I think we have the following choices
A) Instead of writing directly to Kafka from our application we can write our events/messages to some other source (Syslog, File, ?) and then have a separate Java process that reads these sources and writes to Kafka. This is a little annoying since we now have to worry about every machine also running the above separate process to write to Kafka.
B) Work around the above limitations. Auto-discover of brokers is terrible since we don't foresee us adding/removing brokers that frequently. The lack of auto-discover of partitions is definitely a loss since we now have to know which broker/partition to read from at all times. Of course we can just write to Kafka using the above Gems and have our consumers written in another language.
I think this points out the need for a single canonical cross-platform C client lib with support for Zookeeper that could easily wrapped for use in other languages.
It would make it much easier for people using Python, Ruby, Node, Lua, Haskell, Go, OCaml, etc to have such a library that matches the features of the Scala/Java client. Currently writing a client amounts to re-implementing the entire Scala client from scratch, which is why so many clients settle for simple producer/consumer support and skip Zk altogether. On Thu, Aug 29, 2013 at 10:44 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
In 0.8, the producer no longer depends on ZK. It only takes a list of brokers. At LinkedIn, we have a 0.8 C producer implementation and plan to open source it soon.
Jun On Fri, Aug 30, 2013 at 9:00 AM, Travis Brady <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext