Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> one producer and 2 consumers


+
Oleg Ruchovets 2013-04-21, 07:37
+
Philip OToole 2013-04-21, 15:30
+
Oleg Ruchovets 2013-04-21, 15:54
+
Philip OToole 2013-04-21, 16:01
+
Oleg Ruchovets 2013-04-21, 16:16
+
Philip OToole 2013-04-21, 16:22
+
Oleg Ruchovets 2013-04-26, 15:20
+
Jun Rao 2013-04-26, 15:28
+
Chris Curtin 2013-04-26, 15:35
+
Oleg Ruchovets 2013-04-26, 16:22
Copy link to this message
-
Re: one producer and 2 consumers
I'd start on the Wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Index

And the presentations:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations

Also the archives for this list are sometimes informative:
http://mail-archives.apache.org/mod_mbox/kafka-users/

Hope this helps,

Chris
On Fri, Apr 26, 2013 at 12:21 PM, Oleg Ruchovets <[EMAIL PROTECTED]>wrote:

> Thank you Chris. This is exactly what I tried to achieve.
> What is the best source to learn Kafka. Actually my question was a very
> basic and I didn't find any examples or detailed documentation.
> Is there any book or good documentation ?
>
> Thanks
> Oleg.
>
>
> On Fri, Apr 26, 2013 at 6:34 PM, Chris Curtin <[EMAIL PROTECTED]
> >wrote:
>
> > If using the high level consumer you'll need to run two different Groups
> to
> > have both of your consumers receive every message. I'm assuming you want
> > two different types of business logic to process each message? If so you
> > need to treat them as separate groups.
> >
> >
> > On Fri, Apr 26, 2013 at 11:28 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Have you looked at #4 in http://kafka.apache.org/faq.html ?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Fri, Apr 26, 2013 at 8:19 AM, Oleg Ruchovets <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi.
> > > >    I have simple kafka producer/consumer application. I have one
> > producer
> > > > and 2 consumers. consumers has the same code , it is just executed it
> > in
> > > > different threads. For some reason information produced by producer
> > > > consumed only by ONE CONSUMER.Second consumer didn't consumed any
> > > > information. May be I have to add additional configuration
> parameters?
> > > >
> > > > I need is the following:
> > > > 1)Producer produce one event.
> > > > 2)Two consumers and each consumer consumes the  event one time.
> > > >
> > > > For example :
> > > > 1)Producer1 produces message Message1.
> > > > 2)Consumer1  consumes Message1.
> > > > 3)Consumer2 consumes Message1.
> > > >
> > > > What is the way to got such functionality?
> > > >
> > > > Thanks
> > > > Oleg.
> > > >
> > > >
> > > > On Sun, Apr 21, 2013 at 7:21 PM, Philip O'Toole <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > OK, if you want each consumer to process the same data, then simply
> > > > > point each consumer at your Kafka cluster and have each Consumer
> > > > > consume all data. There is no synchronization required between
> those
> > > > > two consumers.
> > > > >
> > > > > In other words, what you want to do is fine. Please read the Kafka
> > > > > design doc if you have not done so:
> > > > >
> > > > > http://kafka.apache.org/design.html
> > > > >
> > > > > Philip
> > > > >
> > > > > On Sun, Apr 21, 2013 at 9:16 AM, Oleg Ruchovets <
> > [EMAIL PROTECTED]>
> > > > > wrote:
> > > > > > I am on POC stage , so I can configure the producer to write in
> > > > different
> > > > > > partitions.
> > > > > > But how it will help me to process the same data with two
> > consumers.
> > > > > >
> > > > > > I try to get such effect:
> > > > > >   I got the data and store it to Kafka.
> > > > > >
> > > > > > I have 2 consumers:
> > > > > >    1) for real time which consumes the data for example every 10
> > > > seconds.
> > > > > >    2) for move data to hdfs - for example every 1 hour.
> > > > > >
> > > > > > But in case I'll use 2 partitions , each consumer process part of
> > the
> > > > > data
> > > > > > (50%). Does it correct?
> > > > > > I need that 2 consumers will produce 100% of the data.
> > > > > >
> > > > > > Please advice.
> > > > > >
> > > > > >
> > > > > > On Sun, Apr 21, 2013 at 12:00 PM, Philip O'Toole <
> > [EMAIL PROTECTED]>
> > > > > wrote:
> > > > > >
> > > > > >> On Sun, Apr 21, 2013 at 8:53 AM, Oleg Ruchovets <
> > > [EMAIL PROTECTED]
> > > > >
> > > > > >> wrote:
> > > > > >> > Hi Philip.
> > > > > >> >    Does it mean to store the same data twice - each time to
> > > > different
> > > > >
 
+
Oleg Ruchovets 2013-04-26, 16:32
+
Chris Curtin 2013-04-26, 16:55