Hi,

I want to dump all changes that happen on ZK to a file.

What I am trying to do is following

   - Set  up some application specific structure on zookeeper.
   - Start a listener or something equivalent that watches on all the nodes.
   - Multiple processes are connected to Zookeeper and they make some
   changes on the ZK as needed
   - The listener will detect the changes and dump the changes to a file in
   some structure
   - Once the run is over there will be a verification tool which will go
   over the change log and do some verifications.

I am not sure if there is already a tool to do this or if it is possible to
get all changes from ZK server logs.

I am aware that if we have a separate listener we may not be able to get all
changes since I might miss changes between receiving a notification and a
change happening before i reset the watch. This may not be the case with ZK
logs/snapshot.

This will not be in production but I want to use it to for testing. Idea is
to decouple verification from simulation so that all types of verification
can be done based on the change log.

Any pointers/suggestions ?

thanks,
Kishore G

you can do this using the logs. there is a log format utility:
LogFormatter that will print the log in a human readable format that
you could use to get this information. (actually you might need to
enhance the tool a bit depending on the detail that you want.)

ben

On Tue, Sep 13, 2011 at 3:53 PM, kishore g <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I want to dump all changes that happen on ZK to a file.
>
> What I am trying to do is following
>
>   - Set  up some application specific structure on zookeeper.
>   - Start a listener or something equivalent that watches on all the nodes.
>   - Multiple processes are connected to Zookeeper and they make some
>   changes on the ZK as needed
>   - The listener will detect the changes and dump the changes to a file in
>   some structure
>   - Once the run is over there will be a verification tool which will go
>   over the change log and do some verifications.
>
> I am not sure if there is already a tool to do this or if it is possible to
> get all changes from ZK server logs.
>
> I am aware that if we have a separate listener we may not be able to get all
> changes since I might miss changes between receiving a notification and a
> change happening before i reset the watch. This may not be the case with ZK
> logs/snapshot.
>
> This will not be in production but I want to use it to for testing. Idea is
> to decouple verification from simulation so that all types of verification
> can be done based on the change log.
>
> Any pointers/suggestions ?
>
> thanks,
> Kishore G
>

Thanks Ben, that's exactly what I was looking for.

On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <[EMAIL PROTECTED]> wrote:

> you can do this using the logs. there is a log format utility:
> LogFormatter that will print the log in a human readable format that
> you could use to get this information. (actually you might need to
> enhance the tool a bit depending on the detail that you want.)
>
> ben
>
> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I want to dump all changes that happen on ZK to a file.
> >
> > What I am trying to do is following
> >
> >   - Set  up some application specific structure on zookeeper.
> >   - Start a listener or something equivalent that watches on all the
> nodes.
> >   - Multiple processes are connected to Zookeeper and they make some
> >   changes on the ZK as needed
> >   - The listener will detect the changes and dump the changes to a file
> in
> >   some structure
> >   - Once the run is over there will be a verification tool which will go
> >   over the change log and do some verifications.
> >
> > I am not sure if there is already a tool to do this or if it is possible
> to
> > get all changes from ZK server logs.
> >
> > I am aware that if we have a separate listener we may not be able to get
> all
> > changes since I might miss changes between receiving a notification and a
> > change happening before i reset the watch. This may not be the case with
> ZK
> > logs/snapshot.
> >
> > This will not be in production but I want to use it to for testing. Idea
> is
> > to decouple verification from simulation so that all types of
> verification
> > can be done based on the change log.
> >
> > Any pointers/suggestions ?
> >
> > thanks,
> > Kishore G
> >
>

Hi Ben,

This only gives me the hugh level information something like

ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
createSession
8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
setData
8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
createSession
...
..
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
setData
8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c create
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
setData
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
setData
8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
setData

How can get more details about the transaction itself. I want to know which
node was created deleted what was set etc. Is this possible ?

thanks,
Kishore G
On Tue, Sep 13, 2011 at 4:47 PM, kishore g <[EMAIL PROTECTED]> wrote:

> Thanks Ben, that's exactly what I was looking for.
>
> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <[EMAIL PROTECTED]> wrote:
>
>> you can do this using the logs. there is a log format utility:
>> LogFormatter that will print the log in a human readable format that
>> you could use to get this information. (actually you might need to
>> enhance the tool a bit depending on the detail that you want.)
>>
>> ben
>>
>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > I want to dump all changes that happen on ZK to a file.
>> >
>> > What I am trying to do is following
>> >
>> >   - Set  up some application specific structure on zookeeper.
>> >   - Start a listener or something equivalent that watches on all the
>> nodes.
>> >   - Multiple processes are connected to Zookeeper and they make some
>> >   changes on the ZK as needed
>> >   - The listener will detect the changes and dump the changes to a file
>> in
>> >   some structure
>> >   - Once the run is over there will be a verification tool which will go
>> >   over the change log and do some verifications.
>> >
>> > I am not sure if there is already a tool to do this or if it is possible
>> to
>> > get all changes from ZK server logs.
>> >
>> > I am aware that if we have a separate listener we may not be able to get
>> all
>> > changes since I might miss changes between receiving a notification and
>> a
>> > change happening before i reset the watch. This may not be the case with
>> ZK
>> > logs/snapshot.
>> >
>> > This will not be in production but I want to use it to for testing. Idea
>> is
>> > to decouple verification from simulation so that all types of
>> verification
>> > can be done based on the change log.
>> >
>> > Any pointers/suggestions ?
>> >
>> > thanks,
>> > Kishore G
>> >
>>
>
>

Hi Ben,

I was able to get more information by changing the LogFormatter code.

Record txn = SerializeUtils.deserializeTxn(iab, hdr);

Based on the txn type, I could cast it to appropriate Txn like SetDataTxn
and was able to get all the information needed.

thanks,
Kishore G

On Tue, Sep 13, 2011 at 5:40 PM, kishore g <[EMAIL PROTECTED]> wrote:

> Hi Ben,
>
> This only gives me the hugh level information something like
>
> ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
> 8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
> createSession
> 8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
> setData
> 8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
> createSession
> ...
> ..
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
> setData
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c
> create
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
> setData
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
> setData
> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
> setData
>
> How can get more details about the transaction itself. I want to know which
> node was created deleted what was set etc. Is this possible ?
>
> thanks,
> Kishore G
>
>
>
> On Tue, Sep 13, 2011 at 4:47 PM, kishore g <[EMAIL PROTECTED]> wrote:
>
>> Thanks Ben, that's exactly what I was looking for.
>>
>> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <[EMAIL PROTECTED]> wrote:
>>
>>> you can do this using the logs. there is a log format utility:
>>> LogFormatter that will print the log in a human readable format that
>>> you could use to get this information. (actually you might need to
>>> enhance the tool a bit depending on the detail that you want.)
>>>
>>> ben
>>>
>>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <[EMAIL PROTECTED]> wrote:
>>> > Hi,
>>> >
>>> > I want to dump all changes that happen on ZK to a file.
>>> >
>>> > What I am trying to do is following
>>> >
>>> >   - Set  up some application specific structure on zookeeper.
>>> >   - Start a listener or something equivalent that watches on all the
>>> nodes.
>>> >   - Multiple processes are connected to Zookeeper and they make some
>>> >   changes on the ZK as needed
>>> >   - The listener will detect the changes and dump the changes to a file
>>> in
>>> >   some structure
>>> >   - Once the run is over there will be a verification tool which will
>>> go
>>> >   over the change log and do some verifications.
>>> >
>>> > I am not sure if there is already a tool to do this or if it is
>>> possible to
>>> > get all changes from ZK server logs.
>>> >
>>> > I am aware that if we have a separate listener we may not be able to
>>> get all
>>> > changes since I might miss changes between receiving a notification and
>>> a
>>> > change happening before i reset the watch. This may not be the case
>>> with ZK
>>> > logs/snapshot.
>>> >
>>> > This will not be in production but I want to use it to for testing.
>>> Idea is
>>> > to decouple verification from simulation so that all types of
>>> verification
>>> > can be done based on the change log.
>>> >
>>> > Any pointers/suggestions ?
>>> >
>>> > thanks,
>>> > Kishore G
>>> >
>>>
>>
>>
>

Consider submitting a patch.

Patrick

On Tue, Sep 13, 2011 at 11:44 PM, kishore g <[EMAIL PROTECTED]> wrote:
> Hi Ben,
>
> I was able to get more information by changing the LogFormatter code.
>
> Record txn = SerializeUtils.deserializeTxn(iab, hdr);
>
> Based on the txn type, I could cast it to appropriate Txn like SetDataTxn
> and was able to get all the information needed.
>
> thanks,
> Kishore G
>
> On Tue, Sep 13, 2011 at 5:40 PM, kishore g <[EMAIL PROTECTED]> wrote:
>
>> Hi Ben,
>>
>> This only gives me the hugh level information something like
>>
>> ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
>> 8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
>> createSession
>> 8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
>> setData
>> 8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
>> createSession
>> ...
>> ..
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
>> setData
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c
>> create
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
>> setData
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
>> setData
>> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
>> setData
>>
>> How can get more details about the transaction itself. I want to know which
>> node was created deleted what was set etc. Is this possible ?
>>
>> thanks,
>> Kishore G
>>
>>
>>
>> On Tue, Sep 13, 2011 at 4:47 PM, kishore g <[EMAIL PROTECTED]> wrote:
>>
>>> Thanks Ben, that's exactly what I was looking for.
>>>
>>> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <[EMAIL PROTECTED]> wrote:
>>>
>>>> you can do this using the logs. there is a log format utility:
>>>> LogFormatter that will print the log in a human readable format that
>>>> you could use to get this information. (actually you might need to
>>>> enhance the tool a bit depending on the detail that you want.)
>>>>
>>>> ben
>>>>
>>>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <[EMAIL PROTECTED]> wrote:
>>>> > Hi,
>>>> >
>>>> > I want to dump all changes that happen on ZK to a file.
>>>> >
>>>> > What I am trying to do is following
>>>> >
>>>> >   - Set  up some application specific structure on zookeeper.
>>>> >   - Start a listener or something equivalent that watches on all the
>>>> nodes.
>>>> >   - Multiple processes are connected to Zookeeper and they make some
>>>> >   changes on the ZK as needed
>>>> >   - The listener will detect the changes and dump the changes to a file
>>>> in
>>>> >   some structure
>>>> >   - Once the run is over there will be a verification tool which will
>>>> go
>>>> >   over the change log and do some verifications.
>>>> >
>>>> > I am not sure if there is already a tool to do this or if it is
>>>> possible to
>>>> > get all changes from ZK server logs.
>>>> >
>>>> > I am aware that if we have a separate listener we may not be able to
>>>> get all
>>>> > changes since I might miss changes between receiving a notification and
>>>> a
>>>> > change happening before i reset the watch. This may not be the case
>>>> with ZK
>>>> > logs/snapshot.
>>>> >
>>>> > This will not be in production but I want to use it to for testing.
>>>> Idea is
>>>> > to decouple verification from simulation so that all types of
>>>> verification
>>>> > can be done based on the change log.
>>>> >
>>>> > Any pointers/suggestions ?
>>>> >
>>>> > thanks,
>>>> > Kishore G
>>>> >
>>>>
>>>
>>>
>>
>

Will do.

https://issues.apache.org/jira/browse/ZOOKEEPER-1183

thanks
Kishore G

On Wed, Sep 14, 2011 at 10:37 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:

> Consider submitting a patch.
>
> Patrick
>
> On Tue, Sep 13, 2011 at 11:44 PM, kishore g <[EMAIL PROTECTED]> wrote:
> > Hi Ben,
> >
> > I was able to get more information by changing the LogFormatter code.
> >
> > Record txn = SerializeUtils.deserializeTxn(iab, hdr);
> >
> > Based on the txn type, I could cast it to appropriate Txn like SetDataTxn
> > and was able to get all the information needed.
> >
> > thanks,
> > Kishore G
> >
> > On Tue, Sep 13, 2011 at 5:40 PM, kishore g <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Ben,
> >>
> >> This only gives me the hugh level information something like
> >>
> >> ZooKeeper Transactional Log File with dbid 0 txnlog format version 2
> >> 8/15/11 1:55:36 PM PDT session 0x131cf1a236f0014 cxid 0x0 zxid 0xf01
> >> createSession
> >> 8/15/11 1:55:57 PM PDT session 0x131cf1a236f0000 cxid 0x55f zxid 0xf02
> >> setData
> >> 8/15/11 1:56:00 PM PDT session 0x131cf1a236f0015 cxid 0x0 zxid 0xf03
> >> createSession
> >> ...
> >> ..
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001c cxid 0x36 zxid 0xf6b
> >> setData
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f0021 cxid 0xa1 zxid 0xf6c
> >> create
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001b cxid 0x3e zxid 0xf6d
> >> setData
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001e cxid 0x3e zxid 0xf6e
> >> setData
> >> 8/15/11 2:00:33 PM PDT session 0x131cf1a236f001d cxid 0x41 zxid 0xf6f
> >> setData
> >>
> >> How can get more details about the transaction itself. I want to know
> which
> >> node was created deleted what was set etc. Is this possible ?
> >>
> >> thanks,
> >> Kishore G
> >>
> >>
> >>
> >> On Tue, Sep 13, 2011 at 4:47 PM, kishore g <[EMAIL PROTECTED]> wrote:
> >>
> >>> Thanks Ben, that's exactly what I was looking for.
> >>>
> >>> On Tue, Sep 13, 2011 at 4:43 PM, Benjamin Reed <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> you can do this using the logs. there is a log format utility:
> >>>> LogFormatter that will print the log in a human readable format that
> >>>> you could use to get this information. (actually you might need to
> >>>> enhance the tool a bit depending on the detail that you want.)
> >>>>
> >>>> ben
> >>>>
> >>>> On Tue, Sep 13, 2011 at 3:53 PM, kishore g <[EMAIL PROTECTED]>
> wrote:
> >>>> > Hi,
> >>>> >
> >>>> > I want to dump all changes that happen on ZK to a file.
> >>>> >
> >>>> > What I am trying to do is following
> >>>> >
> >>>> >   - Set  up some application specific structure on zookeeper.
> >>>> >   - Start a listener or something equivalent that watches on all the
> >>>> nodes.
> >>>> >   - Multiple processes are connected to Zookeeper and they make some
> >>>> >   changes on the ZK as needed
> >>>> >   - The listener will detect the changes and dump the changes to a
> file
> >>>> in
> >>>> >   some structure
> >>>> >   - Once the run is over there will be a verification tool which
> will
> >>>> go
> >>>> >   over the change log and do some verifications.
> >>>> >
> >>>> > I am not sure if there is already a tool to do this or if it is
> >>>> possible to
> >>>> > get all changes from ZK server logs.
> >>>> >
> >>>> > I am aware that if we have a separate listener we may not be able to
> >>>> get all
> >>>> > changes since I might miss changes between receiving a notification
> and
> >>>> a
> >>>> > change happening before i reset the watch. This may not be the case
> >>>> with ZK
> >>>> > logs/snapshot.
> >>>> >
> >>>> > This will not be in production but I want to use it to for testing.
> >>>> Idea is
> >>>> > to decouple verification from simulation so that all types of
> >>>> verification
> >>>> > can be done based on the change log.
> >>>> >
> >>>> > Any pointers/suggestions ?
> >>>> >
> >>>> > thanks,
> >>>> > Kishore G
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
>