Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka 0.8 producer within Play Framework?


+
charlie w 2013-02-02, 00:33
+
charlie w 2013-02-05, 00:13
+
S Ahmed 2013-02-05, 14:38
+
charlie w 2013-02-05, 15:14
Copy link to this message
-
Re: Kafka 0.8 producer within Play Framework?
Hmm interesting...... combining Play and Kafka is a cool combination :) ...

Being completely stateless, Play nodes can go up and down without any
issue. If you're relying on an async producer, however, then you kind of
lose this property of Play, because unpublished messages could be lost in
the event of a node failure (which means the nodes become stateful, within
a small time bound). If the production is sync and non-batched, then the
Play statelessness is maximized, but Kafka's throughput is going to be
severely limited with such settings. Of course, you could have a sync batch
producer, which would be properly stateless and allow Kafka to have a
potentially high throughput, but then Play's throughput will be severely
limited because calls to Kafka will be blocking, and Play is meant to have
few threads that process a high throughput of (shot lived) non-blocking
operations, not a high amount of threads that can each do blocking
operations however they want.

A re-usable async batch producer is what should give you the best possible
throughput for both Play and Kafka, at the expense of risking a bit of data
loss in the event of a failure... In the end, this is pretty much the same
compromise we always make I guess...

--
Felix
On Tue, Feb 5, 2013 at 10:13 AM, charlie w <[EMAIL PROTECTED]> wrote:

> Yes probably, but ATM, I've just been trying to get it to work at all.
>  Fixing issues like that is on the way...
>
> On Tue, Feb 5, 2013 at 7:38 AM, S Ahmed <[EMAIL PROTECTED]> wrote:
> > Shouldn't your producer be at the controller scope?  Instantiating it
> every
> > time is probably nto the correct pattern.  You probably want to use a
> asych
> > producer also right?
> >
> >
> >
> >
> > On Mon, Feb 4, 2013 at 7:12 PM, charlie w <[EMAIL PROTECTED]> wrote:
> >
> >> It seems the issue is related to Scala versions.  I grabbed from
> >> https://github.com/kevinwright/kafka
> >> and built with
> >> sbt "++2.10.0 package"
> >>
> >> I'm able to have my Play app produce messages for Kafka now.
> >>
> >>
> >> On Fri, Feb 1, 2013 at 5:32 PM, charlie w <[EMAIL PROTECTED]>
> wrote:
> >> > Is it possible to have a Kafka 0.8 producer inside a Play Framework
> >> > controller?  (I am new to both Kafka and Play.)
> >> >
> >> > I have managed to get Java code cribbed from the Kafka examples
> >> > (below) to compile and execute within Play, but constructing the
> >> > ProducerConfig object never returns.  I am not finding anything in any
> >> > Play log.
> >> >
> >> > I've also tried this using the Scala console producer as a model and
> >> > had the same result
> >> >
> >> > The Kafka console producer and consumer samples are working for me.
> >> >
> >> > Thanks
> >> > Charlie
> >> >
> >> > code:
> >> >
> >> >
> >> > package controllers;
> >> >
> >> > import java.util.Properties;
> >> >
> >> > import kafka.producer.ProducerConfig;
> >> > import kafka.javaapi.producer.Producer;
> >> > import kafka.producer.KeyedMessage;
> >> >
> >> > public class Hello
> >> > {
> >> >    public static void hello( String v )
> >> >    {
> >> >       Properties props = new Properties();
> >> >       props.put("zk.connect", "127.0.0.1:2181");
> >> >       props.put("serializer.class", "kafka.serializer.StringEncoder");
> >> >       props.put("broker.list", "localhost:9092" );
> >> >
> >> >       play.Logger.info("create config");
> >> >       ProducerConfig config = new ProducerConfig(props);
> >> >
> >> >       play.Logger.info("create producer");
> >> >       Producer<Integer, String> producer = new Producer<Integer,
> >> > String>(config);
> >> >
> >> >       play.Logger.info("yay!");
> >> >
> >> >       KeyedMessage msg = new KeyedMessage<Integer,
> >> > String>("test-topic", "Greetings!");
> >> >
> >> >       producer.send( msg );
> >> >    }
> >> > }
> >>
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB