Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> [Consumer code not working][Kafka Newbie]


Copy link to this message
-
Re: [Consumer code not working][Kafka Newbie]
Will manual offset commit work?

Thanks,

Jun
On Tue, Jan 21, 2014 at 8:11 AM, Abhishek Bhattacharjee <
[EMAIL PROTECTED]> wrote:

> Thanks for the reply.
> Actually in my use-case I need to control the offsets my self so should I
> use SimpleConsumer instead of Group Consumers ?
>
>
> On Tue, Jan 21, 2014 at 9:38 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > "auto.offset.reset" is only used when offsets don't exist in ZK. In your
> > case, the consumer likely already committed the offsets to ZK. So, after
> > restarting, the consumer will resume from where it left off, instead of
> > re-getting everything again. This is the expected behavior during normal
> > operation. If you are testing, you can use a new consumer group.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 21, 2014 at 8:02 AM, Abhishek Bhattacharjee <
> > [EMAIL PROTECTED]> wrote:
> >
> > > I read the faqs and I added "auto.offset.reset" property in the
> > > configuration setting of storm. Then I ran my producer code and then I
> > ran
> > > my consumer code when I ran the consumer code it printed all the
> messages
> > > that were created by producer but after stopping the consumer when I
> ran
> > it
> > > again it didn't show any messages. I think the offset was not reset.
> What
> > > do you think is going wrong ?
> > >
> > > Thanks
> > >
> > >
> > > On Mon, Jan 20, 2014 at 9:42 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Could you check the following FAQ?
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whydoesmyconsumernevergetanydata
> > > > ?
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped,why
> > > > ?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Jan 20, 2014 at 7:22 AM, Abhishek Bhattacharjee <
> > > > [EMAIL PROTECTED]> wrote:
> > > >
> > > > > Sorry I have sent both codes as consumer codes. This is the
> producer
> > > > code.
> > > > >
> > > > > *Producer.java*
> > > > >
> > > > > package kafka.examples;
> > > > >
> > > > >
> > > > > import java.util.Properties;
> > > > > import kafka.producer.KeyedMessage;
> > > > > import kafka.producer.ProducerConfig;
> > > > >
> > > > > public class Producer/* extends Thread*/
> > > > > {
> > > > >   private final kafka.javaapi.producer.Producer<Integer, String>
> > > > producer;
> > > > >   private final String topic;
> > > > >   private final Properties props = new Properties();
> > > > >
> > > > >   public Producer(String topic)
> > > > >   {
> > > > >     props.put("serializer.class",
> "kafka.serializer.StringEncoder");
> > > > >     props.put("metadata.broker.list", "localhost:9092");
> > > > >     // Use random partitioner. Don't need the key type. Just set it
> > to
> > > > > Integer.
> > > > >     // The message is of type String.
> > > > >     producer = new kafka.javaapi.producer.Producer<Integer,
> > String>(new
> > > > > ProducerConfig(props));
> > > > >     this.topic = topic;
> > > > >     System.out.println("Producer at "+this.topic);
> > > > >   }
> > > > >
> > > > >   public void putdata() {
> > > > >     int messageNo = 1;
> > > > >     while(messageNo < 100)
> > > > >     {
> > > > >       String messageStr = new String("Message_" + messageNo);
> > > > >       producer.send(new KeyedMessage<Integer, String>(topic
> > > > ,messageStr));
> > > > >       messageNo = messageNo +1;
> > > > >     }
> > > > >     producer.close();
> > > > >     System.out.println("Producer exit");
> > > > >   }
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > > On Mon, Jan 20, 2014 at 8:46 PM, Abhishek Bhattacharjee <
> > > > > [EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Hello,
> > > > > > I am new to kafka and facing some problem.
> > > > > > My producer code works properly and sends data.
> > > > > > But the consumer is not able to read it.
> > > > > > Here are the codes for Producer and Consumer.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB