I'm trying to track down the exact cause of an ephemeral node being
deleted. I know it could be either (a) the session where the node was
created has expired, or (b) some session, perhaps the same one that created
the node, explicitly deletes the node. Is there a good way to figure out,
client-side or server-side, the actual cause of a node deletion?

Specifically, I have a log message like the following, and I want to know
"where" it came from:

2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
[org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
state:SyncConnected type:NodeDeleted path:/blah/blah/blah for sessionid
0x333a2e75199161b

Also, is the referenced session id the session id of a watch on the deleted
node, or the session id of the session that deleted the node? I'm guessing
the former.

.. Adam

Also, this is running against a 3.3.3 cluster with a 3.3.4 client (to work
around the chroot client bug in 3.3.3, which I was told would only affect
the clients, not the server).

On Thu, Mar 1, 2012 at 6:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:

> I'm trying to track down the exact cause of an ephemeral node being
> deleted. I know it could be either (a) the session where the node was
> created has expired, or (b) some session, perhaps the same one that created
> the node, explicitly deletes the node. Is there a good way to figure out,
> client-side or server-side, the actual cause of a node deletion?
>
> Specifically, I have a log message like the following, and I want to know
> "where" it came from:
>
> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for sessionid
> 0x333a2e75199161b
>
> Also, is the referenced session id the session id of a watch on the
> deleted node, or the session id of the session that deleted the node? I'm
> guessing the former.
>
> .. Adam
>
>

Running your transaction logs through the LogFormatter. You should see
an explicit delete, or the close of the owning session. The
LogFormatter is not awesome though... I have some better versions of
it that I will throw up on GitHub in a bit and link if you're
interested.

On Thu, Mar 1, 2012 at 9:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:
> I'm trying to track down the exact cause of an ephemeral node being
> deleted. I know it could be either (a) the session where the node was
> created has expired, or (b) some session, perhaps the same one that created
> the node, explicitly deletes the node. Is there a good way to figure out,
> client-side or server-side, the actual cause of a node deletion?
>
> Specifically, I have a log message like the following, and I want to know
> "where" it came from:
>
> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for sessionid
> 0x333a2e75199161b
>
> Also, is the referenced session id the session id of a watch on the deleted
> node, or the session id of the session that deleted the node? I'm guessing
> the former.
>
> .. Adam

Actually, I think for this if you just use the LogFormatter that comes
inside ZooKeeper (zookeeper.server.LogFormatter) you should be able to
find the information you need.

C

On Thu, Mar 1, 2012 at 9:47 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:
> Running your transaction logs through the LogFormatter. You should see
> an explicit delete, or the close of the owning session. The
> LogFormatter is not awesome though... I have some better versions of
> it that I will throw up on GitHub in a bit and link if you're
> interested.
>
> On Thu, Mar 1, 2012 at 9:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:
>> I'm trying to track down the exact cause of an ephemeral node being
>> deleted. I know it could be either (a) the session where the node was
>> created has expired, or (b) some session, perhaps the same one that created
>> the node, explicitly deletes the node. Is there a good way to figure out,
>> client-side or server-side, the actual cause of a node deletion?
>>
>> Specifically, I have a log message like the following, and I want to know
>> "where" it came from:
>>
>> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
>> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
>> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for sessionid
>> 0x333a2e75199161b
>>
>> Also, is the referenced session id the session id of a watch on the deleted
>> node, or the session id of the session that deleted the node? I'm guessing
>> the former.
>>
>> .. Adam

Actually, I think for this if you just use the LogFormatter that comes
inside ZooKeeper (zookeeper.server.LogFormatter) you should be able to
find the information you need.

C
>
> On Thu, Mar 1, 2012 at 9:47 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:
>> Running your transaction logs through the LogFormatter. You should see
>> an explicit delete, or the close of the owning session. The
>> LogFormatter is not awesome though... I have some better versions of
>> it that I will throw up on GitHub in a bit and link if you're
>> interested.
>>
>> On Thu, Mar 1, 2012 at 9:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:
>>> I'm trying to track down the exact cause of an ephemeral node being
>>> deleted. I know it could be either (a) the session where the node was
>>> created has expired, or (b) some session, perhaps the same one that created
>>> the node, explicitly deletes the node. Is there a good way to figure out,
>>> client-side or server-side, the actual cause of a node deletion?
>>>
>>> Specifically, I have a log message like the following, and I want to know
>>> "where" it came from:
>>>
>>> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
>>> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
>>> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for sessionid
>>> 0x333a2e75199161b
>>>
>>> Also, is the referenced session id the session id of a watch on the deleted
>>> node, or the session id of the session that deleted the node? I'm guessing
>>> the former.
>>>
>>> .. Adam

That got me some info, looks like some sessions expired exactly then:

3/1/12 5:02:44 PM PST session 0x233a2e751981696 cxid 0x0 zxid 0x1424f05e44
closeSession
3/1/12 5:02:44 PM PST session 0x433a2e7600f15fc cxid 0x0 zxid 0x1424f05e45
closeSession

I'm running multiple ZooKeeper clients in one JVM, that's ok right?

On Thu, Mar 1, 2012 at 7:14 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:

> Actually, I think for this if you just use the LogFormatter that comes
> inside ZooKeeper (zookeeper.server.LogFormatter) you should be able to
> find the information you need.
>
> C
> >
> > On Thu, Mar 1, 2012 at 9:47 PM, Camille Fournier <[EMAIL PROTECTED]>
> wrote:
> >> Running your transaction logs through the LogFormatter. You should see
> >> an explicit delete, or the close of the owning session. The
> >> LogFormatter is not awesome though... I have some better versions of
> >> it that I will throw up on GitHub in a bit and link if you're
> >> interested.
> >>
> >> On Thu, Mar 1, 2012 at 9:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:
> >>> I'm trying to track down the exact cause of an ephemeral node being
> >>> deleted. I know it could be either (a) the session where the node was
> >>> created has expired, or (b) some session, perhaps the same one that
> created
> >>> the node, explicitly deletes the node. Is there a good way to figure
> out,
> >>> client-side or server-side, the actual cause of a node deletion?
> >>>
> >>> Specifically, I have a log message like the following, and I want to
> know
> >>> "where" it came from:
> >>>
> >>> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
> >>> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
> >>> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for sessionid
> >>> 0x333a2e75199161b
> >>>
> >>> Also, is the referenced session id the session id of a watch on the
> deleted
> >>> node, or the session id of the session that deleted the node? I'm
> guessing
> >>> the former.
> >>>
> >>> .. Adam
>

You can do that. All to the same cluster? It can definitely get tricky to
manage depending on what you're using them for.

On Thu, Mar 1, 2012 at 10:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:

> That got me some info, looks like some sessions expired exactly then:
>
> 3/1/12 5:02:44 PM PST session 0x233a2e751981696 cxid 0x0 zxid 0x1424f05e44
> closeSession
> 3/1/12 5:02:44 PM PST session 0x433a2e7600f15fc cxid 0x0 zxid 0x1424f05e45
> closeSession
>
> I'm running multiple ZooKeeper clients in one JVM, that's ok right?
>
> On Thu, Mar 1, 2012 at 7:14 PM, Camille Fournier <[EMAIL PROTECTED]>
> wrote:
>
> > Actually, I think for this if you just use the LogFormatter that comes
> > inside ZooKeeper (zookeeper.server.LogFormatter) you should be able to
> > find the information you need.
> >
> > C
> > >
> > > On Thu, Mar 1, 2012 at 9:47 PM, Camille Fournier <[EMAIL PROTECTED]>
> > wrote:
> > >> Running your transaction logs through the LogFormatter. You should see
> > >> an explicit delete, or the close of the owning session. The
> > >> LogFormatter is not awesome though... I have some better versions of
> > >> it that I will throw up on GitHub in a bit and link if you're
> > >> interested.
> > >>
> > >> On Thu, Mar 1, 2012 at 9:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:
> > >>> I'm trying to track down the exact cause of an ephemeral node being
> > >>> deleted. I know it could be either (a) the session where the node was
> > >>> created has expired, or (b) some session, perhaps the same one that
> > created
> > >>> the node, explicitly deletes the node. Is there a good way to figure
> > out,
> > >>> client-side or server-side, the actual cause of a node deletion?
> > >>>
> > >>> Specifically, I have a log message like the following, and I want to
> > know
> > >>> "where" it came from:
> > >>>
> > >>> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
> > >>> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
> > >>> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for
> sessionid
> > >>> 0x333a2e75199161b
> > >>>
> > >>> Also, is the referenced session id the session id of a watch on the
> > deleted
> > >>> node, or the session id of the session that deleted the node? I'm
> > guessing
> > >>> the former.
> > >>>
> > >>> .. Adam
> >
>

Well I'm instantiating multiple twitter scala-zookeeper-clients (
https://github.com/twitter/scala-zookeeper-client) which each cons up their
own ZooKeeper instances. I haven't patched their code to accept an existing
ZooKeeper instance to share.

So... by tricky, what do you mean?

On Thu, Mar 1, 2012 at 7:44 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:

> You can do that. All to the same cluster? It can definitely get tricky to
> manage depending on what you're using them for.
>
> On Thu, Mar 1, 2012 at 10:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:
>
> > That got me some info, looks like some sessions expired exactly then:
> >
> > 3/1/12 5:02:44 PM PST session 0x233a2e751981696 cxid 0x0 zxid
> 0x1424f05e44
> > closeSession
> > 3/1/12 5:02:44 PM PST session 0x433a2e7600f15fc cxid 0x0 zxid
> 0x1424f05e45
> > closeSession
> >
> > I'm running multiple ZooKeeper clients in one JVM, that's ok right?
> >
> > On Thu, Mar 1, 2012 at 7:14 PM, Camille Fournier <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Actually, I think for this if you just use the LogFormatter that comes
> > > inside ZooKeeper (zookeeper.server.LogFormatter) you should be able to
> > > find the information you need.
> > >
> > > C
> > > >
> > > > On Thu, Mar 1, 2012 at 9:47 PM, Camille Fournier <[EMAIL PROTECTED]
> >
> > > wrote:
> > > >> Running your transaction logs through the LogFormatter. You should
> see
> > > >> an explicit delete, or the close of the owning session. The
> > > >> LogFormatter is not awesome though... I have some better versions of
> > > >> it that I will throw up on GitHub in a bit and link if you're
> > > >> interested.
> > > >>
> > > >> On Thu, Mar 1, 2012 at 9:43 PM, Adam Rosien <[EMAIL PROTECTED]>
> wrote:
> > > >>> I'm trying to track down the exact cause of an ephemeral node being
> > > >>> deleted. I know it could be either (a) the session where the node
> was
> > > >>> created has expired, or (b) some session, perhaps the same one that
> > > created
> > > >>> the node, explicitly deletes the node. Is there a good way to
> figure
> > > out,
> > > >>> client-side or server-side, the actual cause of a node deletion?
> > > >>>
> > > >>> Specifically, I have a log message like the following, and I want
> to
> > > know
> > > >>> "where" it came from:
> > > >>>
> > > >>> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
> > > >>> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
> > > >>> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for
> > sessionid
> > > >>> 0x333a2e75199161b
> > > >>>
> > > >>> Also, is the referenced session id the session id of a watch on the
> > > deleted
> > > >>> node, or the session id of the session that deleted the node? I'm
> > > guessing
> > > >>> the former.
> > > >>>
> > > >>> .. Adam
> > >
> >
>

Depends on what you're doing with the clients. I mean, within the same
process it's likely that you might want to have only one connection to a
given cluster, for reasons like the cause of your original email: if you
have a bunch of ephemeral nodes and they're being closed and you're not
sure, is it because one of my ZK connections within this process was
closed, or did another part of the process actually delete it? If I set a
watch on a given thing, is the intention of that watch to be received only
by the instance of the client within my application that set it, or really
do I want that watch because my application needs to know about it? Etc.
It means you have a weird combination of shared (within the application)
state and distributed (within the ZK) state acting together within the same
system, which can be logically hard to reason about.

C
On Thu, Mar 1, 2012 at 10:47 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:

> Well I'm instantiating multiple twitter scala-zookeeper-clients (
> https://github.com/twitter/scala-zookeeper-client) which each cons up
> their
> own ZooKeeper instances. I haven't patched their code to accept an existing
> ZooKeeper instance to share.
>
> So... by tricky, what do you mean?
>
> On Thu, Mar 1, 2012 at 7:44 PM, Camille Fournier <[EMAIL PROTECTED]>
> wrote:
>
> > You can do that. All to the same cluster? It can definitely get tricky to
> > manage depending on what you're using them for.
> >
> > On Thu, Mar 1, 2012 at 10:43 PM, Adam Rosien <[EMAIL PROTECTED]> wrote:
> >
> > > That got me some info, looks like some sessions expired exactly then:
> > >
> > > 3/1/12 5:02:44 PM PST session 0x233a2e751981696 cxid 0x0 zxid
> > 0x1424f05e44
> > > closeSession
> > > 3/1/12 5:02:44 PM PST session 0x433a2e7600f15fc cxid 0x0 zxid
> > 0x1424f05e45
> > > closeSession
> > >
> > > I'm running multiple ZooKeeper clients in one JVM, that's ok right?
> > >
> > > On Thu, Mar 1, 2012 at 7:14 PM, Camille Fournier <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Actually, I think for this if you just use the LogFormatter that
> comes
> > > > inside ZooKeeper (zookeeper.server.LogFormatter) you should be able
> to
> > > > find the information you need.
> > > >
> > > > C
> > > > >
> > > > > On Thu, Mar 1, 2012 at 9:47 PM, Camille Fournier <
> [EMAIL PROTECTED]
> > >
> > > > wrote:
> > > > >> Running your transaction logs through the LogFormatter. You should
> > see
> > > > >> an explicit delete, or the close of the owning session. The
> > > > >> LogFormatter is not awesome though... I have some better versions
> of
> > > > >> it that I will throw up on GitHub in a bit and link if you're
> > > > >> interested.
> > > > >>
> > > > >> On Thu, Mar 1, 2012 at 9:43 PM, Adam Rosien <[EMAIL PROTECTED]>
> > wrote:
> > > > >>> I'm trying to track down the exact cause of an ephemeral node
> being
> > > > >>> deleted. I know it could be either (a) the session where the node
> > was
> > > > >>> created has expired, or (b) some session, perhaps the same one
> that
> > > > created
> > > > >>> the node, explicitly deletes the node. Is there a good way to
> > figure
> > > > out,
> > > > >>> client-side or server-side, the actual cause of a node deletion?
> > > > >>>
> > > > >>> Specifically, I have a log message like the following, and I want
> > to
> > > > know
> > > > >>> "where" it came from:
> > > > >>>
> > > > >>> 2012-03-01 17:02:44 [main-SendThread(blah:2181)] [DEBUG]
> > > > >>> [org.apache.zookeeper.ClientCnxn         ] - Got WatchedEvent
> > > > >>> state:SyncConnected type:NodeDeleted path:/blah/blah/blah for
> > > sessionid
> > > > >>> 0x333a2e75199161b
> > > > >>>
> > > > >>> Also, is the referenced session id the session id of a watch on
> the
> > > > deleted
> > > > >>> node, or the session id of the session that deleted the node? I'm
> > > > guessing
> > > > >>> the former.
> > > > >>>
> > > > >>> .. Adam
> > > >
> > >
> >
>