Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka Node.js Integration Questions/Advice


+
Christopher Alexander 2012-12-20, 16:47
+
David Arthur 2012-12-20, 16:59
+
Christopher Alexander 2012-12-20, 17:11
+
Jun Rao 2012-12-20, 17:31
+
Radek Gruchalski 2012-12-21, 20:55
+
Apoorva Gaurav 2012-12-22, 04:21
+
Radek Gruchalski 2012-12-22, 16:14
Copy link to this message
-
Re: Kafka Node.js Integration Questions/Advice
Apoorva Gaurav 2012-12-22, 18:00
Thanks Radek,
We also are thinking of Java / Scala for Consumers, for Producers whether
franz-kafka is a good choice?

--
Thanks & Regards,
Apoorva

On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
[EMAIL PROTECTED]> wrote:

> We started using node-kafka before we learned franz-kafka was available.
> In node, franz-kafka would be my preferred choice now. But tbh, our
> consumers are all java. node-kafka does not support consumer settings like
> autooffset.reset and so on (or it is not obvious how to use those).
>
> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
> monitoring tools, which may be helpful if you want to add some scaling
> logic when consumer is lagging.
>
> Our first choice is node too but we're consuming exclusively with java.
>
> Hope this helps a little.
>
> On 22 Dec 2012, at 05:21, Apoorva Gaurav <[EMAIL PROTECTED]> wrote:
>
> > Which is the best ZK based implementation of kafka in node.js. Our use
> case
> > is that a pool of node js http servers will be listening to clients which
> > will send json over http. Using node js we'll do minimal decoration and
> > compression (preferably snappy) and write to brokers. We might also need
> > json to avro conversion but thats not a deal breaker. Consumers will be
> > writing these events to S3 (to begin with we don't plan to maintain HDFS
> > cluster). To begin with we'll have to support a peak load of 50K events /
> > second, average being much less, around 2K events / second. Suggestions
> > please. Is any one using franz-kafka in production. I'm only two days
> into
> > kafka so don't know a lot, but franz-kafka looks clean and easy to work
> > with.
> >
> > If none of the existing node.js implementation is capable of this then we
> > are willing to move to Scala or Java but node.js is the first choice.
> >
> > Thanks & Regards,
> > Apoorva
> >
> > On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
> > [EMAIL PROTECTED]> wrote:
> >
> >> We are using https://github.com/radekg/node-kafka, occasionally pushing
> >> about 2500 messages, 3.5K each / second. No issues so far. Different
> story
> >> with consumers. They are stable but under heavy load we experienced CPU
> >> problems. I am the maintainer of that fork. The fork comes with ZK
> >> integration. Another kafka module is this one:
> >> https://github.com/dannycoates/franz-kafka.
> >>
> >> Kind regards,
> >> Radek Gruchalski
> >> [EMAIL PROTECTED] (mailto:
> [EMAIL PROTECTED])
> >> | [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) |
> >> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> >> 00447889948663
> >>
> >> Confidentiality:
> >> This communication is intended for the above-named person and may be
> >> confidential and/or legally privileged.
> >> If it has come to you in error you must take no action based on it, nor
> >> must you copy or show it to anyone; please delete/destroy and inform the
> >> sender immediately.
> >>
> >>
> >> On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
> >>
> >>> Chris,
> >>>
> >>> Not sure how stable those node.js clients are. In 0.8, we plan to
> >> provide a
> >>> native C version of the producer. A thin node.js layer can potentially
> be
> >>> built on top of that.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
> >>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
> >>>
> >>>> During my due diligence to assess use of Kafka for both our activity
> >> and
> >>>> log message streams, I would like to ask the project committers and
> >>>> community users about using Kafka with Node.js. Yes, I am aware that a
> >>>> Kafka client exists for Node.js (
> >>>> https://github.com/marcuswestin/node-kafka), which has spurred
> further
> >>>> interest by our front-end team. Here are my questions, excuse me if
> >> they
> >>>> seem "noobish".
> >>>>
> >>>> 1. How reliable is the Node.js client (
> >>>
 
+
David Arthur 2012-12-22, 18:17
+
Radek Gruchalski 2012-12-22, 18:38
+
Apoorva Gaurav 2012-12-22, 18:54
+
Christopher Alexander 2012-12-28, 20:27
+
Radek Gruchalski 2012-12-22, 18:37