Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Node-Kafka Client Review and Question


Copy link to this message
-
Node-Kafka Client Review and Question
Christopher Alexander 2013-04-24, 17:20
Hi Everyone,

I just wanted to follow-up on a previous thread concerning our investigation of identifying a stable Node-Kafka client. To date we have tested the following:

1. Franz-Kafka (https://github.com/dannycoates/franz-kafka)
2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka)
3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka)
4. Prozess (v0.3.5, https://github.com/cainus/Prozess)

Results:

1. Could not get Franz-Kafka and Prozess to work. Requires funky dependencies.
2. Node-Kafka, v2.1 was successfully setup but performed less stable than #3.
3. Node-Kafka, v2.3 was successfully setup, exhibited the best performance profile but the consumer is highly inconsistent - specifically, consumer object remained in-memory regardless what we did (i.e. var consumer = undefined after receiving message). Nothing appears to mitigate this and ALL consumed messaged get replayed on reception of a new message.

With this said, is there a Node-Kafka client people are actually using in production that doesn't exhibit the profiles we have seen? We have back-tracked using Node-Kafka (v2.3) to only produce messages and rely on Redis PubSub channels for asynchronous acking of these messages. We would be willing to roll-up our sleeves with the community to develop a much more stable Node-Kafka client.

Kind regards,

Chris Alexander
Chief Technical Architect and Engineer
Gravy, Inc.