Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - one producer and 2 consumers


+
Oleg Ruchovets 2013-04-21, 07:37
+
Philip OToole 2013-04-21, 15:30
+
Oleg Ruchovets 2013-04-21, 15:54
Copy link to this message
-
Re: one producer and 2 consumers
Philip O'Toole 2013-04-21, 16:01
On Sun, Apr 21, 2013 at 8:53 AM, Oleg Ruchovets <[EMAIL PROTECTED]> wrote:
> Hi Philip.
>    Does it mean to store the same data twice - each time to different
> partition? I tried to save data only one time. Using two partitions means
> to store data twice?

No, I mean spreading the data across the two partitions, so 50% goes
in one, and 50% goes in the other. Have your Producer always write to
partition "-1", which will tell Kafka to select a partition at random
for each message.

Then one of the Consumers will consume partition 0, the other partition 1.

> By the way I am using  kafka 0.7.2.
>
> Thanks
> Oleg.
>
>
> On Sun, Apr 21, 2013 at 11:30 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
>> Read the design doc on the Kafka site.
>>
>> The short answer is to use two partitions for your topic.
>>
>> Philip
>>
>> On Apr 21, 2013, at 12:37 AM, Oleg Ruchovets <[EMAIL PROTECTED]> wrote:
>>
>> > Hi,
>> >   I have one producer for kafka and have 2 consumers.
>> > I want to consume produced events to hdfs and storm. Copy to hdfs I will
>> do
>> > every hour but to storm every 10 seconds.
>> >
>> > Question: Is it supported by kafka? Where can I read how to organize 1
>> > producer and 2 consumers?
>> >
>> > Thanks
>> > Oleg.
>>

 
+
Oleg Ruchovets 2013-04-21, 16:16
+
Philip OToole 2013-04-21, 16:22
+
Oleg Ruchovets 2013-04-26, 15:20
+
Jun Rao 2013-04-26, 15:28
+
Chris Curtin 2013-04-26, 15:35
+
Oleg Ruchovets 2013-04-26, 16:22
+
Chris Curtin 2013-04-26, 16:32
+
Oleg Ruchovets 2013-04-26, 16:32
+
Chris Curtin 2013-04-26, 16:55