Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka Node.js Integration Questions/Advice


+
Christopher Alexander 2012-12-20, 16:47
+
David Arthur 2012-12-20, 16:59
+
Christopher Alexander 2012-12-20, 17:11
+
Jun Rao 2012-12-20, 17:31
+
Radek Gruchalski 2012-12-21, 20:55
+
Apoorva Gaurav 2012-12-22, 04:21
+
Radek Gruchalski 2012-12-22, 16:14
+
Apoorva Gaurav 2012-12-22, 18:00
Copy link to this message
-
Re: Kafka Node.js Integration Questions/Advice
David Arthur 2012-12-22, 18:17
FWIW, message production is quite simpler than consumption. It does
not require the same complex coordination as the consumers. Producers
only use ZooKeeper to locate available brokers

Sent from my phone

On Dec 22, 2012, at 1:00 PM, Apoorva Gaurav <[EMAIL PROTECTED]> wrote:

> Thanks Radek,
> We also are thinking of Java / Scala for Consumers, for Producers whether
> franz-kafka is a good choice?
>
> --
> Thanks & Regards,
> Apoorva
>
> On Sat, Dec 22, 2012 at 9:38 PM, Radek Gruchalski <
> [EMAIL PROTECTED]> wrote:
>
>> We started using node-kafka before we learned franz-kafka was available.
>> In node, franz-kafka would be my preferred choice now. But tbh, our
>> consumers are all java. node-kafka does not support consumer settings like
>> autooffset.reset and so on (or it is not obvious how to use those).
>>
>> Afair franz-kafka offers those. Also, java zkconsumer gives you the jmx
>> monitoring tools, which may be helpful if you want to add some scaling
>> logic when consumer is lagging.
>>
>> Our first choice is node too but we're consuming exclusively with java.
>>
>> Hope this helps a little.
>>
>> On 22 Dec 2012, at 05:21, Apoorva Gaurav <[EMAIL PROTECTED]> wrote:
>>
>>> Which is the best ZK based implementation of kafka in node.js. Our use
>> case
>>> is that a pool of node js http servers will be listening to clients which
>>> will send json over http. Using node js we'll do minimal decoration and
>>> compression (preferably snappy) and write to brokers. We might also need
>>> json to avro conversion but thats not a deal breaker. Consumers will be
>>> writing these events to S3 (to begin with we don't plan to maintain HDFS
>>> cluster). To begin with we'll have to support a peak load of 50K events /
>>> second, average being much less, around 2K events / second. Suggestions
>>> please. Is any one using franz-kafka in production. I'm only two days
>> into
>>> kafka so don't know a lot, but franz-kafka looks clean and easy to work
>>> with.
>>>
>>> If none of the existing node.js implementation is capable of this then we
>>> are willing to move to Scala or Java but node.js is the first choice.
>>>
>>> Thanks & Regards,
>>> Apoorva
>>>
>>> On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski <
>>> [EMAIL PROTECTED]> wrote:
>>>
>>>> We are using https://github.com/radekg/node-kafka, occasionally pushing
>>>> about 2500 messages, 3.5K each / second. No issues so far. Different
>> story
>>>> with consumers. They are stable but under heavy load we experienced CPU
>>>> problems. I am the maintainer of that fork. The fork comes with ZK
>>>> integration. Another kafka module is this one:
>>>> https://github.com/dannycoates/franz-kafka.
>>>>
>>>> Kind regards,
>>>> Radek Gruchalski
>>>> [EMAIL PROTECTED] (mailto:
>> [EMAIL PROTECTED])
>>>> | [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED]) |
>>>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])
>>>> 00447889948663
>>>>
>>>> Confidentiality:
>>>> This communication is intended for the above-named person and may be
>>>> confidential and/or legally privileged.
>>>> If it has come to you in error you must take no action based on it, nor
>>>> must you copy or show it to anyone; please delete/destroy and inform the
>>>> sender immediately.
>>>>
>>>>
>>>> On Thursday, 20 December 2012 at 18:31, Jun Rao wrote:
>>>>
>>>>> Chris,
>>>>>
>>>>> Not sure how stable those node.js clients are. In 0.8, we plan to
>>>> provide a
>>>>> native C version of the producer. A thin node.js layer can potentially
>> be
>>>>> built on top of that.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jun
>>>>>
>>>>> On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander <
>>>>> [EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>>>>>
>>>>>> During my due diligence to assess use of Kafka for both our activity
>>>> and
>>>>>> log message streams, I would like to ask the project committers and
>>>>>> community users about using Kafka with Node.js. Yes, I am aware that a

 
+
Radek Gruchalski 2012-12-22, 18:38
+
Apoorva Gaurav 2012-12-22, 18:54
+
Christopher Alexander 2012-12-28, 20:27
+
Radek Gruchalski 2012-12-22, 18:37