Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # dev - Zookeeper 3.3.4 guarantees across sessions from the same client ?


Copy link to this message
-
Re: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Flavio Junqueira 2013-08-03, 00:22
Hi Neha,

Your proposal seems fine to me. When we close a session in PrepRequestProcessor, we line up all ephemerals of the session to be deleted, so as long as the create comes after those deletes, I don't see a problem with checking if the session is closing. Do you want to create a jira so that we can work on it and perhaps other can comment as well?

-Flavio

On Aug 1, 2013, at 7:02 PM,  wrote:

> Hi Neha,
>
> I'm assuming that the Unless we expire a session and delete ephemerals
> atomically, there are only two options I see:
>
> 1- Delete right before expiring the session
> 2- Delete right after expiring the session
>
> Because of timing, we can have the following. With the first, a client
> might observe the delete before the session actually expires, which
> violates our contract. With the second, you may observe an ephemeral znode
> after the session has expired as you have. I would say that the second
> option is correct as long as the ephemerals are eventually deleted, but it
> does have the side-effect you're mentioning.
>
> -Flavio
>
>> -----Original Message-----
>> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
>> Sent: 01 August 2013 02:42
>> To: [EMAIL PROTECTED]
>> Subject: Zookeeper 3.3.4 guarantees across sessions from the same client
> ?
>>
>> The behavior we saw on one of our zookeeper clients is as follows. The
>> session expires on the client, it assumes the ephemeral nodes are
> deleted,
>> so it establishes a new session with zookeeper and tries to re-create
> the
>> ephemeral nodes. However, when it tries to re-create the ephemeral node,
>> zookeeper throws back a NodeExists error code. Now this is legitimate
> during
>> a session disconnect event (since zkclient automatically retries the
> operation
>> and raises a NodeExists error). Also by design, Kafka doesn't have
> multiple
>> clients create the same ephemeral node, so Kafka server assumes the
>> NodeExists is normal. However, after a few seconds zookeeper deletes
> that
>> ephemeral node. So from the client's perspective, even though the client
> has
>> a new valid session, its ephemeral node is gone.
>>
>> After poking at the transaction and log4j logs, I saw that the
> NodeExists was
>> because the zookeeper leader had retained the ephemeral node from the
>> previous expired session. It turns out that it notified the client of
> the session
>> expiration before actually deleting the ephemeral node. It is also worth
>> noting that  the previous session was expired due to a long fsync
> operation
>> on the zookeeper leader. After it returned from the fsync, it had a
> whole
>> bunch of sessions to expire.
>>
>> In this case, it seems that zookeeper should not notify the client that
> the
>> session is expired until the ephemeral node information is actually
> gone.
>> Or maybe I'm not clear on what the guarantees from zookeeper are, across
>> sessions from the same client.
>>
>> Thanks,
>>
>> Neha