Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Getting my Kafka code example working out of the box


+
S L 2013-11-08, 01:41
+
Joe Stein 2013-11-08, 01:51
+
Chris Bedford 2013-11-08, 21:20
+
S L 2013-11-11, 00:34
Copy link to this message
-
Re: Getting my Kafka code example working out of the box
Do you have enough consumers representing the number of partitions you
have?  You should have one consumer within a consumer group for each
partition... Think of it as threads, if you have 5 partitions you need 5
threads... if you only have 3 consumers(threads) then it is possible the
data is on one of the other two partitions that has no one reading from it.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Sun, Nov 10, 2013 at 7:34 PM, S L <[EMAIL PROTECTED]> wrote:

> Thanks for everyone's reply and trying to help me.  I think I got the test
> producer program to work but I don't know why.
>
> Again, using the test code found at
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor
> the producer straight from the website didn't work.  However, when I
> changed the following line from
>
> KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> "page_visits", ip, msg);
> to
> KeyedMessage<String, String> data = new KeyedMessage<String, String>(
> "page_visits", null, msg);
>
> all of a sudden, the consumer gets the messages from my test producer
> program.  Why does this work when the partition is equal to null and not
> work out of the box with the partition equal to ip.  Is there a bug with
> the program that's posted in the link or is there some config that's
> required that's not listed on the links?
>
> Thanks again for everyone's help.  I really appreciate the community
> replying and coming to help me.
>
>
> On Fri, Nov 8, 2013 at 1:20 PM, Chris Bedford <[EMAIL PROTECTED]>
> wrote:
>
> > I had some trouble with maven dependencies when i tried to get a simple
> > round trip test going.   I  worked past those and made my test available
> > here: https://github.com/buildlackey/cep/tree/master/kafka
> >
> > it should run out of the box.
> >
> > -cb
> >
> >
> > On Thu, Nov 7, 2013 at 5:35 PM, S L <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > This might be a really, really simple question but how do I get my test
> > > Kafka program working out of the box?  I followed the directions from
> > > http://kafka.apache.org/documentation.html#quickstart.  I started zk,
> > the
> > > server, the producer and consumer.  I played with the producer, sending
> > > msgs to the consumer, which I saw show up in its terminal.  This was to
> > the
> > > "page_visits" topic, as specified in the example program.
> > >
> > > I then went to
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Exampleand
> > > copied the files into netbeans and built it.  I also added jar files
> > > kafka-assembly-0.8.0-deps.jar and kafka_2.8.0-0.8.0-beta1.jar to make
> the
> > > build work if that makes a difference.
> > >
> > > Anyway, I then ran java -jar myjar.jar.  I expected to see a message
> show
> > > up in the consumer, but nothing shows up.  I sent another msg from the
> > > producer just to make sure the consumer was still connected, which it
> was
> > > b/c the consumer received it.
> > >
> > > I'm at wits end.  I know this is an extremely simple question and
> > > everything looks identical to the links above and should be working.
> > >  However, I'm not getting a message in the consumer.  Much thanks for
> any
> > > help you can give me.  I'm about to ram my head into a wall.
> > >
> >
> >
> >
> > --
> > Chris Bedford
> >
> > Founder & Lead Lackey
> > Build Lackey Labs:  http://buildlackey.com
> > Go Grails!: http://blog.buildlackey.com
> >
>

 
+
Priya Matpadi 2013-11-11, 03:51
+
S L 2013-11-11, 04:45
+
S L 2013-11-11, 04:40