Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka Node.js Integration Questions/Advice


Copy link to this message
-
Re: Kafka Node.js Integration Questions/Advice
Update: Early (1 week) implementation of Node-Kafka has resulted in the following observations:

1. Consumer is unstable.
2. If use of Consumer is mandatory, create the Consumer in application-scope, not request-scope.
3. Attempt to close Consumer on application shutdown. Results of unplanned shutdowns have not been fully evaluated.
4. Producer is stable and works as expected.
5. If producing a message that also necessitates a Kafka response, implement a setTimeout interval to conform with application SLAs.

We will revisit swapping out Node-Kafka with Franz-Kafka once our middleware (Java) buildout is complete. Then we will compare the 2 Kafka Node clients under load and post our results here or in a blog.

Test, test, test. That's the take away here.

----- Original Message -----
From: "Apoorva Gaurav" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Saturday, December 22, 2012 1:54:00 PM
Subject: Re: Kafka Node.js Integration Questions/Advice

Thanks Radek, thanks David.

On Sun, Dec 23, 2012 at 12:08 AM, Radek Gruchalski <
[EMAIL PROTECTED]> wrote:

> That is exactly why we've decided to stick with java. Also support for all
> consumer settings out of the box.
>
> Kind regards,
> Radek Gruchalski
>
> On 22 Dec 2012, at 19:17, David Arthur <[EMAIL PROTECTED]> wrote:
>
> > FWIW, message production is quite simpler than consumption. It does
> > not require the same complex coordination as the consumers. Producers
> > only use ZooKeeper to locate available brokers
> >
> > Sent from my phone
> >
> > On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav <[EMAIL PROTECTED]>
> wrote:
> >
> >> Thanks Radek,
> >> We also are thinking of Java / Scala for Consumers, for Producers
> whether
> >> franz-kafka is a good choice?
> >>
> >> --
> >> Thanks & Regards,
> >> Apoorva
> >>
> >> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >>> We started using node-kafka before we learned franz-kafka was
> available.
> >>> In node, franz-kafka would be my preferred choice now. But tbh, our
> >>> consumers are all java. node-kafka does not support consumer settings
> like
> >>> autooffset.reset and so on (or it is not obvious how to use those).
> >>>
> >>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
> >>> monitoring tools, which may be helpful if you want to add some scaling
> >>> logic when consumer is lagging.
> >>>
> >>> Our first choice is node too but we're consuming exclusively with java.
> >>>
> >>> Hope this helps a little.
> >>>
> >>> On 22 Dec 2012, at 05:21, Apoorva Gaurav <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> Which is the best ZK based implementation of kafka in node.js. Our use
> >>> case
> >>>> is that a pool of node js http servers will be listening to clients
> which
> >>>> will send json over http. Using node js we'll do minimal decoration
> and
> >>>> compression (preferably snappy) and write to brokers. We might also
> need
> >>>> json to avro conversion but thats not a deal breaker. Consumers will
> be
> >>>> writing these events to S3 (to begin with we don't plan to maintain
> HDFS
> >>>> cluster). To begin with we'll have to support a peak load of 50K
> events /
> >>>> second, average being much less, around 2K events / second.
> Suggestions
> >>>> please. Is any one using franz-kafka in production. I'm only two days
> >>> into
> >>>> kafka so don't know a lot, but franz-kafka looks clean and easy to
> work
> >>>> with.
> >>>>
> >>>> If none of the existing node.js implementation is capable of this
> then we
> >>>> are willing to move to Scala or Java but node.js is the first choice.
> >>>>
> >>>> Thanks & Regards,
> >>>> Apoorva
> >>>>
> >>>> On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
> >>>> [EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> We are using https://github.com/radekg/node-kafka, occasionally
> pushing
> >>>>> about 2500 messages, 3.5K each / second. No issues so far. Different
> >>> story
> >>>>> with consumers. They are stable but under heavy load we experienced

Thanks & Regards,
Apoorva

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB