Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - one producer and 2 consumers


Copy link to this message
-
Re: one producer and 2 consumers
Oleg Ruchovets 2013-04-26, 16:32
By the way. What does high level consumer means? Is there other type of
consumers?

Thanks
Oleg.
On Fri, Apr 26, 2013 at 6:34 PM, Chris Curtin <[EMAIL PROTECTED]>wrote:

> If using the high level consumer you'll need to run two different Groups to
> have both of your consumers receive every message. I'm assuming you want
> two different types of business logic to process each message? If so you
> need to treat them as separate groups.
>
>
> On Fri, Apr 26, 2013 at 11:28 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Have you looked at #4 in http://kafka.apache.org/faq.html ?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Apr 26, 2013 at 8:19 AM, Oleg Ruchovets <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi.
> > >    I have simple kafka producer/consumer application. I have one
> producer
> > > and 2 consumers. consumers has the same code , it is just executed it
> in
> > > different threads. For some reason information produced by producer
> > > consumed only by ONE CONSUMER.Second consumer didn't consumed any
> > > information. May be I have to add additional configuration parameters?
> > >
> > > I need is the following:
> > > 1)Producer produce one event.
> > > 2)Two consumers and each consumer consumes the  event one time.
> > >
> > > For example :
> > > 1)Producer1 produces message Message1.
> > > 2)Consumer1  consumes Message1.
> > > 3)Consumer2 consumes Message1.
> > >
> > > What is the way to got such functionality?
> > >
> > > Thanks
> > > Oleg.
> > >
> > >
> > > On Sun, Apr 21, 2013 at 7:21 PM, Philip O'Toole <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > OK, if you want each consumer to process the same data, then simply
> > > > point each consumer at your Kafka cluster and have each Consumer
> > > > consume all data. There is no synchronization required between those
> > > > two consumers.
> > > >
> > > > In other words, what you want to do is fine. Please read the Kafka
> > > > design doc if you have not done so:
> > > >
> > > > http://kafka.apache.org/design.html
> > > >
> > > > Philip
> > > >
> > > > On Sun, Apr 21, 2013 at 9:16 AM, Oleg Ruchovets <
> [EMAIL PROTECTED]>
> > > > wrote:
> > > > > I am on POC stage , so I can configure the producer to write in
> > > different
> > > > > partitions.
> > > > > But how it will help me to process the same data with two
> consumers.
> > > > >
> > > > > I try to get such effect:
> > > > >   I got the data and store it to Kafka.
> > > > >
> > > > > I have 2 consumers:
> > > > >    1) for real time which consumes the data for example every 10
> > > seconds.
> > > > >    2) for move data to hdfs - for example every 1 hour.
> > > > >
> > > > > But in case I'll use 2 partitions , each consumer process part of
> the
> > > > data
> > > > > (50%). Does it correct?
> > > > > I need that 2 consumers will produce 100% of the data.
> > > > >
> > > > > Please advice.
> > > > >
> > > > >
> > > > > On Sun, Apr 21, 2013 at 12:00 PM, Philip O'Toole <
> [EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > >> On Sun, Apr 21, 2013 at 8:53 AM, Oleg Ruchovets <
> > [EMAIL PROTECTED]
> > > >
> > > > >> wrote:
> > > > >> > Hi Philip.
> > > > >> >    Does it mean to store the same data twice - each time to
> > > different
> > > > >> > partition? I tried to save data only one time. Using two
> > partitions
> > > > means
> > > > >> > to store data twice?
> > > > >>
> > > > >> No, I mean spreading the data across the two partitions, so 50%
> goes
> > > > >> in one, and 50% goes in the other. Have your Producer always write
> > to
> > > > >> partition "-1", which will tell Kafka to select a partition at
> > random
> > > > >> for each message.
> > > > >>
> > > > >> Then one of the Consumers will consume partition 0, the other
> > > partition
> > > > 1.
> > > > >>
> > > > >> > By the way I am using  kafka 0.7.2.
> > > > >> >
> > > > >> > Thanks
> > > > >> > Oleg.
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Apr 21, 2013 at 11:30 AM, Philip O'Toole <
> > [EMAIL PROTECTED]
> > > >
> > > > >> wrote: