Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Node-Kafka Client Review and Question

Copy link to this message
Re: Node-Kafka Client Review and Question
We did an examination of the tagged branch but the version was 0.1.7 and its been static for over a year now. I will say that the Node-Kafka (v2.3) producer has been stable however. A previous thread concerning Node-Kafka client development revealed that a C library will be out for 0.8, supporting more stable library client development (hint hint too ;-)

Unfortunately, we are moving rapidly with what works now to get our platform to an MVP state. Hopefully, there won't be too much refactoring when the next gen Node-Kafka client is complete. :(


----- Original Message -----
From: "Taylor Gautier" <[EMAIL PROTECTED]>
Sent: Wednesday, April 24, 2013 4:39:24 PM
Subject: Re: Node-Kafka Client Review and Question

This implementation is what I worked on while at Tagged, which was forked
from Marcus' version, but I don't think it ever merged back to Marcus':


It was in production for about a year when I left Tagged about 6 months
ago.  I know that there were some internal fixes that never made it back
out.  I think the version that is public is pretty stable, but it's hard
for me to know since I am no longer at Tagged and don't have access to the
internal repos to know for sure what fixes are still private and which have
been published back.

That said, I think you should definitely give this version a shot first
before moving on.

Finally, if I were to do it all over again, with about a solid additional
year of node programming experience under my belt, I'd probably rewrite
everything from the interfaces to the implementation.

In my current job there's a strong possibility of us adopting a Node.js +
Kafka implementation, but not for at least a few months, so I wouldn't
expect to be back in the community to work on this for a little while.
 Also, I'm kind of waiting for 0.8 (hint hint ;-)

On Wed, Apr 24, 2013 at 10:45 AM, Christian Carollo <[EMAIL PROTECTED]>wrote:

> Hi Everyone,
> I have been experimenting with the libraries listed below and experienced
> the same problems.
> I have not found any another other node clients.  I am interested in
> finding a node solution as well.
> Happy to contribute on a common solution.
> Christian Carollo
> On Apr 24, 2013, at 10:19 AM, Christopher Alexander <
> > Hi Everyone,
> >
> > I just wanted to follow-up on a previous thread concerning our
> investigation of identifying a stable Node-Kafka client. To date we have
> tested the following:
> >
> > 1. Franz-Kafka (https://github.com/dannycoates/franz-kafka)
> > 2. Node-Kafka (v2.1, https://github.com/radekg/node-kafka)
> > 3. Node-Kafka (v2.3, https://github.com/marcuswestin/node-kafka)
> > 4. Prozess (v0.3.5, https://github.com/cainus/Prozess)
> >
> > Results:
> >
> > 1. Could not get Franz-Kafka and Prozess to work. Requires funky
> dependencies.
> > 2. Node-Kafka, v2.1 was successfully setup but performed less stable
> than #3.
> > 3. Node-Kafka, v2.3 was successfully setup, exhibited the best
> performance profile but the consumer is highly inconsistent - specifically,
> consumer object remained in-memory regardless what we did (i.e. var
> consumer = undefined after receiving message). Nothing appears to mitigate
> this and ALL consumed messaged get replayed on reception of a new message.
> >
> > With this said, is there a Node-Kafka client people are actually using
> in production that doesn't exhibit the profiles we have seen? We have
> back-tracked using Node-Kafka (v2.3) to only produce messages and rely on
> Redis PubSub channels for asynchronous acking of these messages. We would
> be willing to roll-up our sleeves with the community to develop a much more
> stable Node-Kafka client.
> >
> > Kind regards,
> >
> > Chris Alexander
> > Chief Technical Architect and Engineer
> > Gravy, Inc.