Kafka, mail # user - Re: Kafka 0.8 producer within Play Framework? - 2013-02-05, 18:25
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
+
charlie w 2013-02-02, 00:33
+
charlie w 2013-02-05, 00:13
+
S Ahmed 2013-02-05, 14:38
+
charlie w 2013-02-05, 15:14
Copy link to this message
-
Re: Kafka 0.8 producer within Play Framework?
Hmm interesting...... combining Play and Kafka is a cool combination :) ...

Being completely stateless, Play nodes can go up and down without any
issue. If you're relying on an async producer, however, then you kind of
lose this property of Play, because unpublished messages could be lost in
the event of a node failure (which means the nodes become stateful, within
a small time bound). If the production is sync and non-batched, then the
Play statelessness is maximized, but Kafka's throughput is going to be
severely limited with such settings. Of course, you could have a sync batch
producer, which would be properly stateless and allow Kafka to have a
potentially high throughput, but then Play's throughput will be severely
limited because calls to Kafka will be blocking, and Play is meant to have
few threads that process a high throughput of (shot lived) non-blocking
operations, not a high amount of threads that can each do blocking
operations however they want.

A re-usable async batch producer is what should give you the best possible
throughput for both Play and Kafka, at the expense of risking a bit of data
loss in the event of a failure... In the end, this is pretty much the same
compromise we always make I guess...

Felix
On Tue, Feb 5, 2013 at 10:13 AM, charlie w <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB