Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Zookeeper 3.3.4 guarantees across sessions from the same client ?


Copy link to this message
-
RE: Zookeeper 3.3.4 guarantees across sessions from the same client ?
Hi Neha,

Unless we expire a session and delete ephemerals atomically, there are only
two options I see:

1- Delete right before expiring the session
2- Delete right after expiring the session

Because of timing, we can have the following. With the first, a client might
observe the delete before the session actually expires, which violates our
contract. With the second, you may observe an ephemeral znode after the
session has expired as you have. I would say that the second option is
correct as long as the ephemerals are eventually deleted, but it does have
the side-effect you're mentioning.

-Flavio

> -----Original Message-----
> From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
> Sent: 01 August 2013 02:42
> To: [EMAIL PROTECTED]
> Subject: Zookeeper 3.3.4 guarantees across sessions from the same client ?
>
> The behavior we saw on one of our zookeeper clients is as follows. The
> session expires on the client, it assumes the ephemeral nodes are deleted,
> so it establishes a new session with zookeeper and tries to re-create the
> ephemeral nodes. However, when it tries to re-create the ephemeral node,
> zookeeper throws back a NodeExists error code. Now this is legitimate
during
> a session disconnect event (since zkclient automatically retries the
operation
> and raises a NodeExists error). Also by design, Kafka doesn't have
multiple
> clients create the same ephemeral node, so Kafka server assumes the
> NodeExists is normal. However, after a few seconds zookeeper deletes that
> ephemeral node. So from the client's perspective, even though the client
has
> a new valid session, its ephemeral node is gone.
>
> After poking at the transaction and log4j logs, I saw that the NodeExists
was
> because the zookeeper leader had retained the ephemeral node from the
> previous expired session. It turns out that it notified the client of the
session
> expiration before actually deleting the ephemeral node. It is also worth
> noting that  the previous session was expired due to a long fsync
operation
> on the zookeeper leader. After it returned from the fsync, it had a whole
> bunch of sessions to expire.
>
> In this case, it seems that zookeeper should not notify the client that
the
> session is expired until the ephemeral node information is actually gone.
> Or maybe I'm not clear on what the guarantees from zookeeper are, across
> sessions from the same client.
>
> Thanks,
>
> Neha