Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # dev - Zookeeper 3.3.4 guarantees across sessions from the same client ?


Copy link to this message
-
Re: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Flavio Junqueira 2013-08-03, 14:00
I was actually thinking that if the create comes after the deletes, it should already observe that the znode has been deleted, so there is no point in adding the check. I'm also wondering if the scenario you have observed has been resolved in the 3.4 branch. If you can reproduce it, then it would be good to check.

-Flavio
On Aug 3, 2013, at 1:22 AM, Flavio Junqueira <[EMAIL PROTECTED]> wrote:

> Hi Neha,
>
> Your proposal seems fine to me. When we close a session in PrepRequestProcessor, we line up all ephemerals of the session to be deleted, so as long as the create comes after those deletes, I don't see a problem with checking if the session is closing. Do you want to create a jira so that we can work on it and perhaps other can comment as well?
>
> -Flavio
>
> On Aug 1, 2013, at 7:02 PM,  wrote:
>
>> Hi Neha,
>>
>> I'm assuming that the Unless we expire a session and delete ephemerals
>> atomically, there are only two options I see:
>>
>> 1- Delete right before expiring the session
>> 2- Delete right after expiring the session
>>
>> Because of timing, we can have the following. With the first, a client
>> might observe the delete before the session actually expires, which
>> violates our contract. With the second, you may observe an ephemeral znode
>> after the session has expired as you have. I would say that the second
>> option is correct as long as the ephemerals are eventually deleted, but it
>> does have the side-effect you're mentioning.
>>
>> -Flavio
>>
>>> -----Original Message-----
>>> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
>>> Sent: 01 August 2013 02:42
>>> To: [EMAIL PROTECTED]
>>> Subject: Zookeeper 3.3.4 guarantees across sessions from the same client
>> ?
>>>
>>> The behavior we saw on one of our zookeeper clients is as follows. The
>>> session expires on the client, it assumes the ephemeral nodes are
>> deleted,
>>> so it establishes a new session with zookeeper and tries to re-create
>> the
>>> ephemeral nodes. However, when it tries to re-create the ephemeral node,
>>> zookeeper throws back a NodeExists error code. Now this is legitimate
>> during
>>> a session disconnect event (since zkclient automatically retries the
>> operation
>>> and raises a NodeExists error). Also by design, Kafka doesn't have
>> multiple
>>> clients create the same ephemeral node, so Kafka server assumes the
>>> NodeExists is normal. However, after a few seconds zookeeper deletes
>> that
>>> ephemeral node. So from the client's perspective, even though the client
>> has
>>> a new valid session, its ephemeral node is gone.
>>>
>>> After poking at the transaction and log4j logs, I saw that the
>> NodeExists was
>>> because the zookeeper leader had retained the ephemeral node from the
>>> previous expired session. It turns out that it notified the client of
>> the session
>>> expiration before actually deleting the ephemeral node. It is also worth
>>> noting that  the previous session was expired due to a long fsync
>> operation
>>> on the zookeeper leader. After it returned from the fsync, it had a
>> whole
>>> bunch of sessions to expire.
>>>
>>> In this case, it seems that zookeeper should not notify the client that
>> the
>>> session is expired until the ephemeral node information is actually
>> gone.
>>> Or maybe I'm not clear on what the guarantees from zookeeper are, across
>>> sessions from the same client.
>>>
>>> Thanks,
>>>
>>> Neha
>