Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Is Avro right for me?

Copy link to this message
Re: Is Avro right for me?
On 28 May 2013, at 23:38, Mark <[EMAIL PROTECTED]> wrote:
> I actually looked into Kafka quite some time ago and I think we passed on it because it didn't have much ruby support (That may have changed by now).

Ruby support unfortunately continues to be weak. I actually wrote a brand new Kafka producer client for Ruby (not yet open source), but from Kafka 0.7 to 0.8 the wire protocol is changing (as replication & high availability features are added), which means I'm going to have to re-do it.

The Kafka project officially supports producers using the JVM or the C client (the latter could be embedded in Ruby using FFI), and consumers using the JVM client only.

However, that doesn't rule out other languages. LinkedIn actually operates a large service that consumes Kafka using the JVM client, writes the Avro messages to stdout as JSON, and then pipes that into a python process' stdin. Works surprisingly well!