Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - one producer and 2 consumers


+
Oleg Ruchovets 2013-04-21, 07:37
+
Philip OToole 2013-04-21, 15:30
+
Oleg Ruchovets 2013-04-21, 15:54
+
Philip OToole 2013-04-21, 16:01
Copy link to this message
-
Re: one producer and 2 consumers
Oleg Ruchovets 2013-04-21, 16:16
I am on POC stage , so I can configure the producer to write in different
partitions.
But how it will help me to process the same data with two consumers.

I try to get such effect:
  I got the data and store it to Kafka.

I have 2 consumers:
   1) for real time which consumes the data for example every 10 seconds.
   2) for move data to hdfs - for example every 1 hour.

But in case I'll use 2 partitions , each consumer process part of the data
(50%). Does it correct?
I need that 2 consumers will produce 100% of the data.

Please advice.
On Sun, Apr 21, 2013 at 12:00 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:

> On Sun, Apr 21, 2013 at 8:53 AM, Oleg Ruchovets <[EMAIL PROTECTED]>
> wrote:
> > Hi Philip.
> >    Does it mean to store the same data twice - each time to different
> > partition? I tried to save data only one time. Using two partitions means
> > to store data twice?
>
> No, I mean spreading the data across the two partitions, so 50% goes
> in one, and 50% goes in the other. Have your Producer always write to
> partition "-1", which will tell Kafka to select a partition at random
> for each message.
>
> Then one of the Consumers will consume partition 0, the other partition 1.
>
> > By the way I am using  kafka 0.7.2.
> >
> > Thanks
> > Oleg.
> >
> >
> > On Sun, Apr 21, 2013 at 11:30 AM, Philip O'Toole <[EMAIL PROTECTED]>
> wrote:
> >
> >> Read the design doc on the Kafka site.
> >>
> >> The short answer is to use two partitions for your topic.
> >>
> >> Philip
> >>
> >> On Apr 21, 2013, at 12:37 AM, Oleg Ruchovets <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > Hi,
> >> >   I have one producer for kafka and have 2 consumers.
> >> > I want to consume produced events to hdfs and storm. Copy to hdfs I
> will
> >> do
> >> > every hour but to storm every 10 seconds.
> >> >
> >> > Question: Is it supported by kafka? Where can I read how to organize 1
> >> > producer and 2 consumers?
> >> >
> >> > Thanks
> >> > Oleg.
> >>
>

 
+
Philip OToole 2013-04-21, 16:22
+
Oleg Ruchovets 2013-04-26, 15:20
+
Jun Rao 2013-04-26, 15:28
+
Chris Curtin 2013-04-26, 15:35
+
Oleg Ruchovets 2013-04-26, 16:22
+
Chris Curtin 2013-04-26, 16:32
+
Oleg Ruchovets 2013-04-26, 16:32
+
Chris Curtin 2013-04-26, 16:55