During my due diligence to assess use of Kafka for both our activity and log message streams, I would like to ask the project committers and community users about using Kafka with Node.js. Yes, I am aware that a Kafka client exists for Node.js (https://github.com/marcuswestin/node-kafka), which has spurred further interest by our front-end team. Here are my questions, excuse me if they seem "noobish".
1. How reliable is the Node.js client (https://github.com/marcuswestin/node-kafka) in production applications? If there are issues, what are they (the GitHub repo currently lists none)? 2. To support real-time activity streams within Node.js, what is the recommended consumer polling interval? 3. General advise observations on integrating a front-end based Node.js application with Kafka mediated messaging.
On 12/20/12 11:46 AM, Christopher Alexander wrote: Just FYI, there is another node.js library https://github.com/cainus/Prozess. I have no experience with either, so I cannot say how reliable they are. What kind of data velocity do you expect? You should only have to poll if your consumer catches up to the broker and there's no more data. Blocking/polling behavior of the consumer depends entirely on the client implementation.
Thanks David. Yes, I am aware of the Prozess Node lib also. I forgot to include it in my posting. Good catch!
From: "David Arthur" <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Sent: Thursday, December 20, 2012 11:58:45 AM Subject: Re: Kafka Node.js Integration Questions/Advice On 12/20/12 11:46 AM, Christopher Alexander wrote: Just FYI, there is another node.js library https://github.com/cainus/Prozess. I have no experience with either, so I cannot say how reliable they are. What kind of data velocity do you expect? You should only have to poll if your consumer catches up to the broker and there's no more data. Blocking/polling behavior of the consumer depends entirely on the client implementation.
Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately. On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
Which is the best ZK based implementation of kafka in node.js. Our use case is that a pool of node js http servers will be listening to clients which will send json over http. Using node js we'll do minimal decoration and compression (preferably snappy) and write to brokers. We might also need json to avro conversion but thats not a deal breaker. Consumers will be writing these events to S3 (to begin with we don't plan to maintain HDFS cluster). To begin with we'll have to support a peak load of 50K events / second, average being much less, around 2K events / second. Suggestions please. Is any one using franz-kafka in production. I'm only two days into kafka so don't know a lot, but franz-kafka looks clean and easy to work with.
If none of the existing node.js implementation is capable of this then we are willing to move to Scala or Java but node.js is the first choice.
Thanks & Regards, Apoorva
On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski < [EMAIL PROTECTED]> wrote:
We started using node-kafka before we learned franz-kafka was available. In node, franz-kafka would be my preferred choice now. But tbh, our consumers are all java. node-kafka does not support consumer settings like autooffset.reset and so on (or it is not obvious how to use those).
Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx monitoring tools, which may be helpful if you want to add some scaling logic when consumer is lagging.
Our first choice is node too but we're consuming exclusively with java.
Hope this helps a little.
On 22 Dec 2012, at 05:21, Apoorva Gaurav <[EMAIL PROTECTED]> wrote:
Update: Early (1 week) implementation of Node-Kafka has resulted in the following observations:
1. Consumer is unstable. 2. If use of Consumer is mandatory, create the Consumer in application-scope, not request-scope. 3. Attempt to close Consumer on application shutdown. Results of unplanned shutdowns have not been fully evaluated. 4. Producer is stable and works as expected. 5. If producing a message that also necessitates a Kafka response, implement a setTimeout interval to conform with application SLAs.
We will revisit swapping out Node-Kafka with Franz-Kafka once our middleware (Java) buildout is complete. Then we will compare the 2 Kafka Node clients under load and post our results here or in a blog.