Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer offset not saved in zk


Copy link to this message
-
Re: consumer offset not saved in zk
You don't want to override the default configs. Also, seems like something
else is wrong with your setup ? Could you share the log4j logs of your
consumer ? Meanwhile, can you try if you can use the console consumer
successfully ?

Thanks,
Neha
On Thu, May 23, 2013 at 2:31 PM, rk vishu <[EMAIL PROTECTED]> wrote:

> My ZK directory listing is as below. Looks like offsets path is not even
> created.
>
> zk: localhost:2181(CONNECTED) 0] ls /
> [hadoop-ha, hbase, zookeeper, consumers, controller, storm, brokers,
> controller_epoch]
> [zk: localhost:2181(CONNECTED) 1] ls /consumers
> [1, das-service]
> [zk: localhost:2181(CONNECTED) 2] ls /consumers/1
> [owners, ids]
> [zk: localhost:2181(CONNECTED) 3] ls /consumers/1/offsets
> Node does not exist: /consumers/1/offsets
> [zk: localhost:2181(CONNECTED) 4]
>
>
> On Thu, May 23, 2013 at 7:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > You are looking at the wrong path in ZK. The correct path for consumer
> > offset is /consumers/[groupId]/offsets/[topic]/[partitionId] -> long
> > (offset). For more details on our ZK layout, see
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, May 22, 2013 at 11:21 PM, rk vishu <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hello All,
> > >
> > > I recently started experimenting Kafka for my usecase. I am running 0.8
> > in
> > > two node kafka setup.
> > >
> > > I produced 20messages using a java program(1 partition with 2 replicas)
> > and
> > > I am running the consumer code as given in the example
> > > https://cwiki.apache.org/KAFKA/consumer-group-example.html. Consumer
> > > consumes all the messages and after 10sec program shuts down. Based on
> > the
> > > configuration in the example, i am expecting that consumer offsets will
> > be
> > > saved on ZK. If i start the program again, i should not be consuming
> the
> > > same messages again. But i am seeing different behavior. Messages are
> > > getting replayed. I am not seeing any update in ZK also.
> > >
> > > [zk: localhost:2181(CONNECTED) 13] get /consumers/1
> > > null
> > > cZxid = 0x8009ff0ee
> > > ctime = Wed May 22 16:59:21 PDT 2013
> > > mZxid = 0x8009ff0ee
> > > mtime = Wed May 22 16:59:21 PDT 2013
> > > pZxid = 0x8009ff0f4
> > > cversion = 2
> > > dataVersion = 0
> > > aclVersion = 0
> > > ephemeralOwner = 0x0
> > > dataLength = 0
> > > numChildren = 2
> > >
> > >
> > > Could any one of you explain me what could be the issue?
> > >
> > > Ravi
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB