Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Is Avro right for me?


Copy link to this message
-
Re: Is Avro right for me?
On 28 May 2013, at 23:38, Mark <[EMAIL PROTECTED]> wrote:
> I actually looked into Kafka quite some time ago and I think we passed on it because it didn't have much ruby support (That may have changed by now).

Ruby support unfortunately continues to be weak. I actually wrote a brand new Kafka producer client for Ruby (not yet open source), but from Kafka 0.7 to 0.8 the wire protocol is changing (as replication & high availability features are added), which means I'm going to have to re-do it.

The Kafka project officially supports producers using the JVM or the C client (the latter could be embedded in Ruby using FFI), and consumers using the JVM client only.

However, that doesn't rule out other languages. LinkedIn actually operates a large service that consumes Kafka using the JVM client, writes the Avro messages to stdout as JSON, and then pipes that into a python process' stdin. Works surprisingly well!

Martin
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB