Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka Node.js Integration Questions/Advice


Copy link to this message
-
Re: Kafka Node.js Integration Questions/Advice
Thanks Radek, thanks David.

On Sun, Dec 23, 2012 at 12:08 AM, Radek Gruchalski <
[EMAIL PROTECTED]> wrote:

> That is exactly why we've decided to stick with java. Also support for all
> consumer settings out of the box.
>
> Kind regards,
> Radek Gruchalski
>
> On 22 Dec 2012, at 19:17, David Arthur <[EMAIL PROTECTED]> wrote:
>
> > FWIW, message production is quite simpler than consumption. It does
> > not require the same complex coordination as the consumers. Producers
> > only use ZooKeeper to locate available brokers
> >
> > Sent from my phone
> >
> > On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav <[EMAIL PROTECTED]>
> wrote:
> >
> >> Thanks Radek,
> >> We also are thinking of Java / Scala for Consumers, for Producers
> whether
> >> franz-kafka is a good choice?
> >>
> >> --
> >> Thanks & Regards,
> >> Apoorva
> >>
> >> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
> >> [EMAIL PROTECTED]> wrote:
> >>
> >>> We started using node-kafka before we learned franz-kafka was
> available.
> >>> In node, franz-kafka would be my preferred choice now. But tbh, our
> >>> consumers are all java. node-kafka does not support consumer settings
> like
> >>> autooffset.reset and so on (or it is not obvious how to use those).
> >>>
> >>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
> >>> monitoring tools, which may be helpful if you want to add some scaling
> >>> logic when consumer is lagging.
> >>>
> >>> Our first choice is node too but we're consuming exclusively with java.
> >>>
> >>> Hope this helps a little.
> >>>
> >>> On 22 Dec 2012, at 05:21, Apoorva Gaurav <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> Which is the best ZK based implementation of kafka in node.js. Our use
> >>> case
> >>>> is that a pool of node js http servers will be listening to clients
> which
> >>>> will send json over http. Using node js we'll do minimal decoration
> and
> >>>> compression (preferably snappy) and write to brokers. We might also
> need
> >>>> json to avro conversion but thats not a deal breaker. Consumers will
> be
> >>>> writing these events to S3 (to begin with we don't plan to maintain
> HDFS
> >>>> cluster). To begin with we'll have to support a peak load of 50K
> events /
> >>>> second, average being much less, around 2K events / second.
> Suggestions
> >>>> please. Is any one using franz-kafka in production. I'm only two days
> >>> into
> >>>> kafka so don't know a lot, but franz-kafka looks clean and easy to
> work
> >>>> with.
> >>>>
> >>>> If none of the existing node.js implementation is capable of this
> then we
> >>>> are willing to move to Scala or Java but node.js is the first choice.
> >>>>
> >>>> Thanks & Regards,
> >>>> Apoorva
> >>>>
> >>>> On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
> >>>> [EMAIL PROTECTED]> wrote:
> >>>>
> >>>>> We are using https://github.com/radekg/node-kafka, occasionally
> pushing
> >>>>> about 2500 messages, 3.5K each / second. No issues so far. Different
> >>> story
> >>>>> with consumers. They are stable but under heavy load we experienced
> CPU
> >>>>> problems. I am the maintainer of that fork. The fork comes with ZK
> >>>>> integration. Another kafka module is this one:
> >>>>> https://github.com/dannycoates/franz-kafka.
> >>>>>
> >>>>> Kind regards,
> >>>>> Radek Gruchalski
> >>>>> [EMAIL PROTECTED] (mailto:
> >>> [EMAIL PROTECTED])
> >>>>> | [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) |
> >>>>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
> >>>>> 00447889948663
> >>>>>
> >>>>> Confidentiality:
> >>>>> This communication is intended for the above-named person and may be
> >>>>> confidential and/or legally privileged.
> >>>>> If it has come to you in error you must take no action based on it,
> nor
> >>>>> must you copy or show it to anyone; please delete/destroy and inform
> the
> >>>>> sender immediately.
> >>>>>
> >>>>>
> >>>>> On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:

Thanks & Regards,
Apoorva