Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Consumer resetting if zookeeper folder deleted


Copy link to this message
-
Re: Consumer resetting if zookeeper folder deleted
navneet sharma 2012-05-08, 08:35
Hi Neha,

if i delete the zookeper folder, then i can replicate this problem.
Actually, that folder was getting created in "/tmp" folder and it was
getting deleted each day. Now i changed the location and i am seeing that
the exception is gone.

Basically, i was just checking if i restart the consumer, say after 2-3
days, where it will start consuming from. My assumption was that this
information should be part of broker/server but looks like its part of
zookeeper and thats why i observed these exceptions.

Attaching logs for your reference.

Thanks,
Navneet Sharma

On Tue, May 8, 2012 at 12:06 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> Is it an expected behaviour?
>
> Not really. That exception means that the Zookeeper server your client is
> connected to, is trying to send a response to the client's "ping", but
> failed, since probably the client disconnected from the server already.
> I'm not sure I understand what led to this. Could you attach the entire
> kafka + zookeeper log here ?
>
> Thanks,
> Neha
>
>
> On Mon, May 7, 2012 at 8:18 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > ZK stores critical data such as topics/partitions on each broker, in
> > addition to consumer offsets. So ZK data needs to be protected.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, May 7, 2012 at 4:21 AM, navneet sharma
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hi,
> > >
> > > I observed that if zookeeper folder is deleted, consumer is trying to
> > read
> > > the messages it read in previous runs. And there are lots of exceptions
> > > seen on zookeeper console as well::
> > >
> > > java.nio.channels.CancelledKeyException
> > >    at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
> > >    at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)
> > > [2012-05-07 15:57:35,569] ERROR Unexpected Exception:
> > > (org.apache.zookeeper.server.NIOServerCnxn)
> > >
> > >
> > > Is it an expected behaviour? Is zookeeper directory need to be
> protected
> > to
> > > avoid consumer falling off on offsets?
> > >
> > > Thanks,
> > > Navneet Sharma
> > >
> >
>