Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> one producer and 2 consumers


+
Oleg Ruchovets 2013-04-21, 07:37
+
Philip OToole 2013-04-21, 15:30
+
Oleg Ruchovets 2013-04-21, 15:54
Copy link to this message
-
Re: one producer and 2 consumers
On Sun, Apr 21, 2013 at 8:53 AM, Oleg Ruchovets <[EMAIL PROTECTED]> wrote:
> Hi Philip.
>    Does it mean to store the same data twice - each time to different
> partition? I tried to save data only one time. Using two partitions means
> to store data twice?

No, I mean spreading the data across the two partitions, so 50% goes
in one, and 50% goes in the other. Have your Producer always write to
partition "-1", which will tell Kafka to select a partition at random
for each message.

Then one of the Consumers will consume partition 0, the other partition 1.

> By the way I am using  kafka 0.7.2.
>
> Thanks
> Oleg.
>
>
> On Sun, Apr 21, 2013 at 11:30 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
>> Read the design doc on the Kafka site.
>>
>> The short answer is to use two partitions for your topic.
>>
>> Philip
>>
>> On Apr 21, 2013, at 12:37 AM, Oleg Ruchovets <[EMAIL PROTECTED]> wrote:
>>
>> > Hi,
>> >   I have one producer for kafka and have 2 consumers.
>> > I want to consume produced events to hdfs and storm. Copy to hdfs I will
>> do
>> > every hour but to storm every 10 seconds.
>> >
>> > Question: Is it supported by kafka? Where can I read how to organize 1
>> > producer and 2 consumers?
>> >
>> > Thanks
>> > Oleg.
>>

 
+
Oleg Ruchovets 2013-04-21, 16:16
+
Philip OToole 2013-04-21, 16:22
+
Oleg Ruchovets 2013-04-26, 15:20
+
Jun Rao 2013-04-26, 15:28
+
Chris Curtin 2013-04-26, 15:35
+
Oleg Ruchovets 2013-04-26, 16:22
+
Chris Curtin 2013-04-26, 16:32
+
Oleg Ruchovets 2013-04-26, 16:32
+
Chris Curtin 2013-04-26, 16:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB