Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer offset not saved in zk


Copy link to this message
-
Re: consumer offset not saved in zk
You don't want to override the default configs. Also, seems like something
else is wrong with your setup ? Could you share the log4j logs of your
consumer ? Meanwhile, can you try if you can use the console consumer
successfully ?

Thanks,
Neha
On Thu, May 23, 2013 at 2:31 PM, rk vishu <[EMAIL PROTECTED]> wrote:

> My ZK directory listing is as below. Looks like offsets path is not even
> created.
>
> zk: localhost:2181(CONNECTED) 0] ls /
> [hadoop-ha, hbase, zookeeper, consumers, controller, storm, brokers,
> controller_epoch]
> [zk: localhost:2181(CONNECTED) 1] ls /consumers
> [1, das-service]
> [zk: localhost:2181(CONNECTED) 2] ls /consumers/1
> [owners, ids]
> [zk: localhost:2181(CONNECTED) 3] ls /consumers/1/offsets
> Node does not exist: /consumers/1/offsets
> [zk: localhost:2181(CONNECTED) 4]
>
>
> On Thu, May 23, 2013 at 7:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > You are looking at the wrong path in ZK. The correct path for consumer
> > offset is /consumers/[groupId]/offsets/[topic]/[partitionId] -> long
> > (offset). For more details on our ZK layout, see
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, May 22, 2013 at 11:21 PM, rk vishu <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hello All,
> > >
> > > I recently started experimenting Kafka for my usecase. I am running 0.8
> > in
> > > two node kafka setup.
> > >
> > > I produced 20messages using a java program(1 partition with 2 replicas)
> > and
> > > I am running the consumer code as given in the example
> > > https://cwiki.apache.org/KAFKA/consumer-group-example.html. Consumer
> > > consumes all the messages and after 10sec program shuts down. Based on
> > the
> > > configuration in the example, i am expecting that consumer offsets will
> > be
> > > saved on ZK. If i start the program again, i should not be consuming
> the
> > > same messages again. But i am seeing different behavior. Messages are
> > > getting replayed. I am not seeing any update in ZK also.
> > >
> > > [zk: localhost:2181(CONNECTED) 13] get /consumers/1
> > > null
> > > cZxid = 0x8009ff0ee
> > > ctime = Wed May 22 16:59:21 PDT 2013
> > > mZxid = 0x8009ff0ee
> > > mtime = Wed May 22 16:59:21 PDT 2013
> > > pZxid = 0x8009ff0f4
> > > cversion = 2
> > > dataVersion = 0
> > > aclVersion = 0
> > > ephemeralOwner = 0x0
> > > dataLength = 0
> > > numChildren = 2
> > >
> > >
> > > Could any one of you explain me what could be the issue?
> > >
> > > Ravi
> > >
> >
>