Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Getting my Kafka code example working out of the box


Copy link to this message
-
Re: Getting my Kafka code example working out of the box
If you have 5 partitions and 3 consumers in one consumer group, the
consumers will balance all 5 partitions such that 2 consumers will get data
from 2 partitions each and 1 consumer will get data from remaining 5th
partition.
If you have 3 partitions and 5 consumers in one consumer group, 3 consumers
will become leader for the 3 partitions(1 partition per consumer) and 2
consumers will starve. Refer to
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example.
On Sun, Nov 10, 2013 at 5:28 PM, Joe Stein <[EMAIL PROTECTED]> wrote:

> Do you have enough consumers representing the number of partitions you
> have?  You should have one consumer within a consumer group for each
> partition... Think of it as threads, if you have 5 partitions you need 5
> threads... if you only have 3 consumers(threads) then it is possible the
> data is on one of the other two partitions that has no one reading from it.
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
>
> On Sun, Nov 10, 2013 at 7:34 PM, S L <[EMAIL PROTECTED]> wrote:
>
> > Thanks for everyone's reply and trying to help me.  I think I got the
> test
> > producer program to work but I don't know why.
> >
> > Again, using the test code found at
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor
> > the producer straight from the website didn't work.  However, when I
> > changed the following line from
> >
> > KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> > "page_visits", ip, msg);
> > to
> > KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> > "page_visits", null, msg);
> >
> > all of a sudden, the consumer gets the messages from my test producer
> > program.  Why does this work when the partition is equal to null and not
> > work out of the box with the partition equal to ip.  Is there a bug with
> > the program that's posted in the link or is there some config that's
> > required that's not listed on the links?
> >
> > Thanks again for everyone's help.  I really appreciate the community
> > replying and coming to help me.
> >
> >
> > On Fri, Nov 8, 2013 at 1:20 PM, Chris Bedford <[EMAIL PROTECTED]>
> > wrote:
> >
> > > I had some trouble with maven dependencies when i tried to get a simple
> > > round trip test going.   I  worked past those and made my test
> available
> > > here: https://github.com/buildlackey/cep/tree/master/kafka
> > >
> > > it should run out of the box.
> > >
> > > -cb
> > >
> > >
> > > On Thu, Nov 7, 2013 at 5:35 PM, S L <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi,
> > > >
> > > > This might be a really, really simple question but how do I get my
> test
> > > > Kafka program working out of the box?  I followed the directions from
> > > > http://kafka.apache.org/documentation.html#quickstart.  I started
> zk,
> > > the
> > > > server, the producer and consumer.  I played with the producer,
> sending
> > > > msgs to the consumer, which I saw show up in its terminal.  This was
> to
> > > the
> > > > "page_visits" topic, as specified in the example program.
> > > >
> > > > I then went to
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Exampleand
> > > > copied the files into netbeans and built it.  I also added jar files
> > > > kafka-assembly-0.8.0-deps.jar and kafka_2.8.0-0.8.0-beta1.jar to make
> > the
> > > > build work if that makes a difference.
> > > >
> > > > Anyway, I then ran java -jar myjar.jar.  I expected to see a message
> > show
> > > > up in the consumer, but nothing shows up.  I sent another msg from
> the
> > > > producer just to make sure the consumer was still connected, which it
> > was
> > > > b/c the consumer received it.
> > > >
> > > > I'm at wits end.  I know this is an extremely simple question and

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB