Hi,

I observed that if zookeeper folder is deleted, consumer is trying to read
the messages it read in previous runs. And there are lots of exceptions
seen on zookeeper console as well::

java.nio.channels.CancelledKeyException
    at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
    at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
    at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
    at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
    at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
    at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)
[2012-05-07 15:57:35,569] ERROR Unexpected Exception:
(org.apache.zookeeper.server.NIOServerCnxn)
Is it an expected behaviour? Is zookeeper directory need to be protected to
avoid consumer falling off on offsets?

Thanks,
Navneet Sharma

ZK stores critical data such as topics/partitions on each broker, in
addition to consumer offsets. So ZK data needs to be protected.

Thanks,

Jun

On Mon, May 7, 2012 at 4:21 AM, navneet sharma
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> I observed that if zookeeper folder is deleted, consumer is trying to read
> the messages it read in previous runs. And there are lots of exceptions
> seen on zookeeper console as well::
>
> java.nio.channels.CancelledKeyException
>    at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
>    at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
>    at
>
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
>    at
>
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
>    at
>
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
>    at
>
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)
> [2012-05-07 15:57:35,569] ERROR Unexpected Exception:
> (org.apache.zookeeper.server.NIOServerCnxn)
>
>
> Is it an expected behaviour? Is zookeeper directory need to be protected to
> avoid consumer falling off on offsets?
>
> Thanks,
> Navneet Sharma
>

>> Is it an expected behaviour?

Not really. That exception means that the Zookeeper server your client is
connected to, is trying to send a response to the client's "ping", but
failed, since probably the client disconnected from the server already.
I'm not sure I understand what led to this. Could you attach the entire
kafka + zookeeper log here ?

Thanks,
Neha
On Mon, May 7, 2012 at 8:18 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> ZK stores critical data such as topics/partitions on each broker, in
> addition to consumer offsets. So ZK data needs to be protected.
>
> Thanks,
>
> Jun
>
> On Mon, May 7, 2012 at 4:21 AM, navneet sharma
> <[EMAIL PROTECTED]>wrote:
>
> > Hi,
> >
> > I observed that if zookeeper folder is deleted, consumer is trying to
> read
> > the messages it read in previous runs. And there are lots of exceptions
> > seen on zookeeper console as well::
> >
> > java.nio.channels.CancelledKeyException
> >    at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
> >    at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
> >    at
> >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
> >    at
> >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
> >    at
> >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
> >    at
> >
> >
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)
> > [2012-05-07 15:57:35,569] ERROR Unexpected Exception:
> > (org.apache.zookeeper.server.NIOServerCnxn)
> >
> >
> > Is it an expected behaviour? Is zookeeper directory need to be protected
> to
> > avoid consumer falling off on offsets?
> >
> > Thanks,
> > Navneet Sharma
> >
>

Hi Neha,

if i delete the zookeper folder, then i can replicate this problem.
Actually, that folder was getting created in "/tmp" folder and it was
getting deleted each day. Now i changed the location and i am seeing that
the exception is gone.

Basically, i was just checking if i restart the consumer, say after 2-3
days, where it will start consuming from. My assumption was that this
information should be part of broker/server but looks like its part of
zookeeper and thats why i observed these exceptions.

Attaching logs for your reference.

Thanks,
Navneet Sharma

On Tue, May 8, 2012 at 12:06 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> Is it an expected behaviour?
>
> Not really. That exception means that the Zookeeper server your client is
> connected to, is trying to send a response to the client's "ping", but
> failed, since probably the client disconnected from the server already.
> I'm not sure I understand what led to this. Could you attach the entire
> kafka + zookeeper log here ?
>
> Thanks,
> Neha
>
>
> On Mon, May 7, 2012 at 8:18 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > ZK stores critical data such as topics/partitions on each broker, in
> > addition to consumer offsets. So ZK data needs to be protected.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, May 7, 2012 at 4:21 AM, navneet sharma
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Hi,
> > >
> > > I observed that if zookeeper folder is deleted, consumer is trying to
> > read
> > > the messages it read in previous runs. And there are lots of exceptions
> > > seen on zookeeper console as well::
> > >
> > > java.nio.channels.CancelledKeyException
> > >    at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
> > >    at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
> > >    at
> > >
> > >
> >
> org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)
> > > [2012-05-07 15:57:35,569] ERROR Unexpected Exception:
> > > (org.apache.zookeeper.server.NIOServerCnxn)
> > >
> > >
> > > Is it an expected behaviour? Is zookeeper directory need to be
> protected
> > to
> > > avoid consumer falling off on offsets?
> > >
> > > Thanks,
> > > Navneet Sharma
> > >
> >
>

Navneet, Zookeeper is a persistent storage system. If you delete its
data and are not running in distributed mode it will start-over,
right? I think the answer is not to delete the Zookeeper data, right?

-Jay

On Tue, May 8, 2012 at 1:35 AM, navneet sharma
<[EMAIL PROTECTED]> wrote:
> Hi Neha,
>
> if i delete the zookeper folder, then i can replicate this problem.
> Actually, that folder was getting created in "/tmp" folder and it was
> getting deleted each day. Now i changed the location and i am seeing that
> the exception is gone.
>
> Basically, i was just checking if i restart the consumer, say after 2-3
> days, where it will start consuming from. My assumption was that this
> information should be part of broker/server but looks like its part of
> zookeeper and thats why i observed these exceptions.
>
> Attaching logs for your reference.
>
> Thanks,
> Navneet Sharma
>
>
> On Tue, May 8, 2012 at 12:06 AM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
>>
>> >> Is it an expected behaviour?
>>
>> Not really. That exception means that the Zookeeper server your client is
>> connected to, is trying to send a response to the client's "ping", but
>> failed, since probably the client disconnected from the server already.
>> I'm not sure I understand what led to this. Could you attach the entire
>> kafka + zookeeper log here ?
>>
>> Thanks,
>> Neha
>>
>>
>> On Mon, May 7, 2012 at 8:18 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>
>> > ZK stores critical data such as topics/partitions on each broker, in
>> > addition to consumer offsets. So ZK data needs to be protected.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Mon, May 7, 2012 at 4:21 AM, navneet sharma
>> > <[EMAIL PROTECTED]>wrote:
>> >
>> > > Hi,
>> > >
>> > > I observed that if zookeeper folder is deleted, consumer is trying to
>> > read
>> > > the messages it read in previous runs. And there are lots of
>> > > exceptions
>> > > seen on zookeeper console as well::
>> > >
>> > > java.nio.channels.CancelledKeyException
>> > >    at
>> > > sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
>> > >    at
>> > > sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
>> > >    at
>> > >
>> > >
>> >
>> > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
>> > >    at
>> > >
>> > >
>> >
>> > org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
>> > >    at
>> > >
>> > >
>> >
>> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
>> > >    at
>> > >
>> > >
>> >
>> > org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)
>> > > [2012-05-07 15:57:35,569] ERROR Unexpected Exception:
>> > > (org.apache.zookeeper.server.NIOServerCnxn)
>> > >
>> > >
>> > > Is it an expected behaviour? Is zookeeper directory need to be
>> > > protected
>> > to
>> > > avoid consumer falling off on offsets?
>> > >
>> > > Thanks,
>> > > Navneet Sharma
>> > >
>> >
>
>

Navneet,

  In the design document, this is described fairly well.  Basically,
it is up to the consumers to keep track of where they are in the
consumption of each topic/partition.  The supplied consumers use
zookeeper for this.  Check out the 2nd paragraph named "Consumer
State" in http://incubator.apache.org/kafka/design.html

Ed

On Tue, May 8, 2012 at 11:38 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> Navneet, Zookeeper is a persistent storage system. If you delete its
> data and are not running in distributed mode it will start-over,
> right? I think the answer is not to delete the Zookeeper data, right?
>
> -Jay
>
> On Tue, May 8, 2012 at 1:35 AM, navneet sharma
> <[EMAIL PROTECTED]> wrote:
>> Hi Neha,
>>
>> if i delete the zookeper folder, then i can replicate this problem.
>> Actually, that folder was getting created in "/tmp" folder and it was
>> getting deleted each day. Now i changed the location and i am seeing that
>> the exception is gone.
>>
>> Basically, i was just checking if i restart the consumer, say after 2-3
>> days, where it will start consuming from. My assumption was that this
>> information should be part of broker/server but looks like its part of
>> zookeeper and thats why i observed these exceptions.
>>
>> Attaching logs for your reference.
>>
>> Thanks,
>> Navneet Sharma
>>
>>
>> On Tue, May 8, 2012 at 12:06 AM, Neha Narkhede <[EMAIL PROTECTED]>
>> wrote:
>>>
>>> >> Is it an expected behaviour?
>>>
>>> Not really. That exception means that the Zookeeper server your client is
>>> connected to, is trying to send a response to the client's "ping", but
>>> failed, since probably the client disconnected from the server already.
>>> I'm not sure I understand what led to this. Could you attach the entire
>>> kafka + zookeeper log here ?
>>>
>>> Thanks,
>>> Neha
>>>
>>>
>>> On Mon, May 7, 2012 at 8:18 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>>>
>>> > ZK stores critical data such as topics/partitions on each broker, in
>>> > addition to consumer offsets. So ZK data needs to be protected.
>>> >
>>> > Thanks,
>>> >
>>> > Jun
>>> >
>>> > On Mon, May 7, 2012 at 4:21 AM, navneet sharma
>>> > <[EMAIL PROTECTED]>wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > I observed that if zookeeper folder is deleted, consumer is trying to
>>> > read
>>> > > the messages it read in previous runs. And there are lots of
>>> > > exceptions
>>> > > seen on zookeeper console as well::
>>> > >
>>> > > java.nio.channels.CancelledKeyException
>>> > >    at
>>> > > sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
>>> > >    at
>>> > > sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
>>> > >    at
>>> > >
>>> > >
>>> >
>>> > org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
>>> > >    at
>>> > >
>>> > >
>>> >
>>> > org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
>>> > >    at
>>> > >
>>> > >
>>> >
>>> > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
>>> > >    at
>>> > >
>>> > >
>>> >
>>> > org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:135)
>>> > > [2012-05-07 15:57:35,569] ERROR Unexpected Exception:
>>> > > (org.apache.zookeeper.server.NIOServerCnxn)
>>> > >
>>> > >
>>> > > Is it an expected behaviour? Is zookeeper directory need to be
>>> > > protected
>>> > to
>>> > > avoid consumer falling off on offsets?
>>> > >
>>> > > Thanks,
>>> > > Navneet Sharma
>>> > >
>>> >
>>
>>

Navneet,

Thanks for clarifying that. That would explain the exceptions.

Jay,

>> If you delete its data and are not running in distributed mode it will
start-over, right?

This is like erasing the zookeeper server's memory and starting it up.
Works fine, except it doesn't know the session that the Kafka client is
using to talk to it to checkpoint the offsets.
So it logs a bunch of errors, basically rejecting all requests from the
Kafka consumer.

So, if you happen to delete zookeeper data, you also need to restart ALL
the clients that were talking to it. Or else, it will fill up logs with
such rejection error messages.

Thanks,
Neha

On Tue, May 8, 2012 at 9:52 AM, Edward Smith <[EMAIL PROTECTED]> wrote:

> Navneet,
>
>  In the design document, this is described fairly well.  Basically,
> it is up to the consumers to keep track of where they are in the
> consumption of each topic/partition.  The supplied consumers use
> zookeeper for this.  Check out the 2nd paragraph named "Consumer
> State" in http://incubator.apache.org/kafka/design.html
>
> Ed
>
> On Tue, May 8, 2012 at 11:38 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > Navneet, Zookeeper is a persistent storage system. If you delete its
> > data and are not running in distributed mode it will start-over,
> > right? I think the answer is not to delete the Zookeeper data, right?
> >
> > -Jay
> >
> > On Tue, May 8, 2012 at 1:35 AM, navneet sharma
> > <[EMAIL PROTECTED]> wrote:
> >> Hi Neha,
> >>
> >> if i delete the zookeper folder, then i can replicate this problem.
> >> Actually, that folder was getting created in "/tmp" folder and it was
> >> getting deleted each day. Now i changed the location and i am seeing
> that
> >> the exception is gone.
> >>
> >> Basically, i was just checking if i restart the consumer, say after 2-3
> >> days, where it will start consuming from. My assumption was that this
> >> information should be part of broker/server but looks like its part of
> >> zookeeper and thats why i observed these exceptions.
> >>
> >> Attaching logs for your reference.
> >>
> >> Thanks,
> >> Navneet Sharma
> >>
> >>
> >> On Tue, May 8, 2012 at 12:06 AM, Neha Narkhede <[EMAIL PROTECTED]
> >
> >> wrote:
> >>>
> >>> >> Is it an expected behaviour?
> >>>
> >>> Not really. That exception means that the Zookeeper server your client
> is
> >>> connected to, is trying to send a response to the client's "ping", but
> >>> failed, since probably the client disconnected from the server already.
> >>> I'm not sure I understand what led to this. Could you attach the entire
> >>> kafka + zookeeper log here ?
> >>>
> >>> Thanks,
> >>> Neha
> >>>
> >>>
> >>> On Mon, May 7, 2012 at 8:18 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>>
> >>> > ZK stores critical data such as topics/partitions on each broker, in
> >>> > addition to consumer offsets. So ZK data needs to be protected.
> >>> >
> >>> > Thanks,
> >>> >
> >>> > Jun
> >>> >
> >>> > On Mon, May 7, 2012 at 4:21 AM, navneet sharma
> >>> > <[EMAIL PROTECTED]>wrote:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > I observed that if zookeeper folder is deleted, consumer is trying
> to
> >>> > read
> >>> > > the messages it read in previous runs. And there are lots of
> >>> > > exceptions
> >>> > > seen on zookeeper console as well::
> >>> > >
> >>> > > java.nio.channels.CancelledKeyException
> >>> > >    at
> >>> > > sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
> >>> > >    at
> >>> > > sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59)
> >>> > >    at
> >>> > >
> >>> > >
> >>> >
> >>> >
> org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:418)
> >>> > >    at
> >>> > >
> >>> > >
> >>> >
> >>> >
> org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1509)
> >>> > >    at
> >>> > >
> >>> > >
> >>> >
> >>> >
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:171)
> >>> > >    at
> >>> > >
> >