Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Getting my Kafka code example working out of the box


Copy link to this message
-
Re: Getting my Kafka code example working out of the box
Do you have enough consumers representing the number of partitions you
have?  You should have one consumer within a consumer group for each
partition... Think of it as threads, if you have 5 partitions you need 5
threads... if you only have 3 consumers(threads) then it is possible the
data is on one of the other two partitions that has no one reading from it.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Sun, Nov 10, 2013 at 7:34 PM, S L <[EMAIL PROTECTED]> wrote:

> Thanks for everyone's reply and trying to help me.  I think I got the test
> producer program to work but I don't know why.
>
> Again, using the test code found at
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor
> the producer straight from the website didn't work.  However, when I
> changed the following line from
>
> KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> "page_visits", ip, msg);
> to
> KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> "page_visits", null, msg);
>
> all of a sudden, the consumer gets the messages from my test producer
> program.  Why does this work when the partition is equal to null and not
> work out of the box with the partition equal to ip.  Is there a bug with
> the program that's posted in the link or is there some config that's
> required that's not listed on the links?
>
> Thanks again for everyone's help.  I really appreciate the community
> replying and coming to help me.
>
>
> On Fri, Nov 8, 2013 at 1:20 PM, Chris Bedford <[EMAIL PROTECTED]>
> wrote:
>
> > I had some trouble with maven dependencies when i tried to get a simple
> > round trip test going.   I  worked past those and made my test available
> > here: https://github.com/buildlackey/cep/tree/master/kafka
> >
> > it should run out of the box.
> >
> > -cb
> >
> >
> > On Thu, Nov 7, 2013 at 5:35 PM, S L <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > This might be a really, really simple question but how do I get my test
> > > Kafka program working out of the box?  I followed the directions from
> > > http://kafka.apache.org/documentation.html#quickstart.  I started zk,
> > the
> > > server, the producer and consumer.  I played with the producer, sending
> > > msgs to the consumer, which I saw show up in its terminal.  This was to
> > the
> > > "page_visits" topic, as specified in the example program.
> > >
> > > I then went to
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Exampleand
> > > copied the files into netbeans and built it.  I also added jar files
> > > kafka-assembly-0.8.0-deps.jar and kafka_2.8.0-0.8.0-beta1.jar to make
> the
> > > build work if that makes a difference.
> > >
> > > Anyway, I then ran java -jar myjar.jar.  I expected to see a message
> show
> > > up in the consumer, but nothing shows up.  I sent another msg from the
> > > producer just to make sure the consumer was still connected, which it
> was
> > > b/c the consumer received it.
> > >
> > > I'm at wits end.  I know this is an extremely simple question and
> > > everything looks identical to the links above and should be working.
> > >  However, I'm not getting a message in the consumer.  Much thanks for
> any
> > > help you can give me.  I'm about to ram my head into a wall.
> > >
> >
> >
> >
> > --
> > Chris Bedford
> >
> > Founder & Lead Lackey
> > Build Lackey Labs:  http://buildlackey.com
> > Go Grails!: http://blog.buildlackey.com
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB