Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Getting my Kafka code example working out of the box


Copy link to this message
-
Re: Getting my Kafka code example working out of the box
Priya Matpadi 2013-11-11, 03:51
If you have 5 partitions and 3 consumers in one consumer group, the
consumers will balance all 5 partitions such that 2 consumers will get data
from 2 partitions each and 1 consumer will get data from remaining 5th
partition.
If you have 3 partitions and 5 consumers in one consumer group, 3 consumers
will become leader for the 3 partitions(1 partition per consumer) and 2
consumers will starve. Refer to
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example.
On Sun, Nov 10, 2013 at 5:28 PM, Joe Stein <[EMAIL PROTECTED]> wrote:

> Do you have enough consumers representing the number of partitions you
> have?  You should have one consumer within a consumer group for each
> partition... Think of it as threads, if you have 5 partitions you need 5
> threads... if you only have 3 consumers(threads) then it is possible the
> data is on one of the other two partitions that has no one reading from it.
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
>
> On Sun, Nov 10, 2013 at 7:34 PM, S L <[EMAIL PROTECTED]> wrote:
>
> > Thanks for everyone's reply and trying to help me.  I think I got the
> test
> > producer program to work but I don't know why.
> >
> > Again, using the test code found at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor
> > the producer straight from the website didn't work.  However, when I
> > changed the following line from
> >
> > KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> > "page_visits", ip, msg);
> > to
> > KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> > "page_visits", null, msg);
> >
> > all of a sudden, the consumer gets the messages from my test producer
> > program.  Why does this work when the partition is equal to null and not
> > work out of the box with the partition equal to ip.  Is there a bug with
> > the program that's posted in the link or is there some config that's
> > required that's not listed on the links?
> >
> > Thanks again for everyone's help.  I really appreciate the community
> > replying and coming to help me.
> >
> >
> > On Fri, Nov 8, 2013 at 1:20 PM, Chris Bedford <[EMAIL PROTECTED]>
> > wrote:
> >
> > > I had some trouble with maven dependencies when i tried to get a simple
> > > round trip test going.   I  worked past those and made my test
> available
> > > here: https://github.com/buildlackey/cep/tree/master/kafka
> > >
> > > it should run out of the box.
> > >
> > > -cb
> > >
> > >
> > > On Thu, Nov 7, 2013 at 5:35 PM, S L <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi,
> > > >
> > > > This might be a really, really simple question but how do I get my
> test
> > > > Kafka program working out of the box?  I followed the directions from
> > > > http://kafka.apache.org/documentation.html#quickstart.  I started
> zk,
> > > the
> > > > server, the producer and consumer.  I played with the producer,
> sending
> > > > msgs to the consumer, which I saw show up in its terminal.  This was
> to
> > > the
> > > > "page_visits" topic, as specified in the example program.
> > > >
> > > > I then went to
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Exampleand
> > > > copied the files into netbeans and built it.  I also added jar files
> > > > kafka-assembly-0.8.0-deps.jar and kafka_2.8.0-0.8.0-beta1.jar to make
> > the
> > > > build work if that makes a difference.
> > > >
> > > > Anyway, I then ran java -jar myjar.jar.  I expected to see a message
> > show
> > > > up in the consumer, but nothing shows up.  I sent another msg from
> the
> > > > producer just to make sure the consumer was still connected, which it
> > was
> > > > b/c the consumer received it.
> > > >
> > > > I'm at wits end.  I know this is an extremely simple question and