Hi all:

I'm working on a distributed system these days, and need more conditional
write support on Zookeeper. Now the zookeeper only support modifing, delete
or set, node data with a version number represent the current version of the
node. I need modification on the condition of other nodes. For e.g. I want
to set the node data of /node to A, if the node data of /node1 is B and the
node data of /node2 is C. Should we support this kind of interface?

thanks
--
With Regards!

Ye, Qian

Hi Qian,
   There have been discussions on multiple transaction in zookeeper:

 http://www.mail-archive.com/[EMAIL PROTECTED]/msg08315.html

Which I think might help this case. There have been some discussions on this
lately but not much progress. You can start a discussion on the list to see
what folks feel about multiple transactions. I think if we are to support
something like what you mention it should be via multiple transactions
wherein a success would mean a success for all the transactions that are
part of a multiple transaction.

Thanks
mahadev

On 12/9/10 4:19 AM, "Qian Ye" <[EMAIL PROTECTED]> wrote:

> Hi all:
>
> I'm working on a distributed system these days, and need more conditional
> write support on Zookeeper. Now the zookeeper only support modifing, delete
> or set, node data with a version number represent the current version of the
> node. I need modification on the condition of other nodes. For e.g. I want
> to set the node data of /node to A, if the node data of /node1 is B and the
> node data of /node2 is C. Should we support this kind of interface?
>
> thanks
> --
> With Regards!
>
> Ye, Qian
>

Qian,

Depending on your situation, you can implement something like this now with
the ZK primitives.

In particular,

    - get the current version v_a of A
    - test the values of B and C
    - if the condition on B and C is met, update A with required version v_a

You may want to retry the whole thing if you get an exception on the update
of A.

This does a safe test and set operation, but does not allow for the
potential of atomically updating multiple znodes in one operation.  A
special case solution to that is to put all objects that may need to be
updated together in the same znode content.  That is clearly not a general
solution, but it is often possible.

On Thu, Dec 9, 2010 at 4:19 AM, Qian Ye <[EMAIL PROTECTED]> wrote:

> Hi all:
>
> I'm working on a distributed system these days, and need more conditional
> write support on Zookeeper. Now the zookeeper only support modifing, delete
> or set, node data with a version number represent the current version of
> the
> node. I need modification on the condition of other nodes. For e.g. I want
> to set the node data of /node to A, if the node data of /node1 is B and the
> node data of /node2 is C. Should we support this kind of interface?
>
> thanks
> --
> With Regards!
>
> Ye, Qian
>

Hi Ted:

The solution you mentioned works in some situation, but not mine. Because,
in the third step, after you checking the condition on B and C, the value on
B and C still might be modified before you update value on A. The key point
is that with the current ZK primitives, you cannot lock node A when u are
updating node B.

A possible solution based on current ZK primitives for this scenario is that
create a extra node for each data node to play as the lock. So the update on
A can be protected by this kind of lock. However, this implementation will
bring in much complexity. For example, how to prevent deadlock in some
abnormal situations.

What's more, I think this kind of conditional write support is simpler than
multiple transactions. Multiple transactions can be built with this kind of
support. The link is broken?
http://www.mail-archive.com/[EMAIL PROTECTED]/msg08315.html
On Fri, Dec 10, 2010 at 7:33 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Qian,
>
> Depending on your situation, you can implement something like this now with
> the ZK primitives.
>
> In particular,
>
>    - get the current version v_a of A
>    - test the values of B and C
>    - if the condition on B and C is met, update A with required version v_a
>
> You may want to retry the whole thing if you get an exception on the update
> of A.
>
> This does a safe test and set operation, but does not allow for the
> potential of atomically updating multiple znodes in one operation.  A
> special case solution to that is to put all objects that may need to be
> updated together in the same znode content.  That is clearly not a general
> solution, but it is often possible.
>
> On Thu, Dec 9, 2010 at 4:19 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>
> > Hi all:
> >
> > I'm working on a distributed system these days, and need more conditional
> > write support on Zookeeper. Now the zookeeper only support modifing,
> delete
> > or set, node data with a version number represent the current version of
> > the
> > node. I need modification on the condition of other nodes. For e.g. I
> want
> > to set the node data of /node to A, if the node data of /node1 is B and
> the
> > node data of /node2 is C. Should we support this kind of interface?
> >
> > thanks
> > --
> > With Regards!
> >
> > Ye, Qian
> >
>

--
With Regards!

Ye, Qian

Yes.  I can imagine that being able to write several variables, specifying
the version of each would
be useful.  But not yet supported.

On Fri, Dec 10, 2010 at 12:01 AM, Qian Ye <[EMAIL PROTECTED]> wrote:

> What's more, I think this kind of conditional write support is simpler than
> multiple transactions. Multiple transactions can be built with this kind of
> support. The link is broken?
>

I have read the mails on
http://www.mail-archive.com/[EMAIL PROTECTED]/msg08315.html,
and here is some further thinking about this issue.

The interface like

zoo_multi_test_and_set(List<int> versions, List<string> znodes,
List<byte[]> data）

can solve the problem I mentioned before, and some relavant issues, like
hard for programmers to use, as mentioned in mail-archive, should be paid
attention to. I think we can move small step first, that is, provide
interface like

zoo_multi_test_and_set(List<int> versions, List<string> znodes, byte[]
data, string znode）
The API test versions of several different znodes before set one znode, and
if the client want to set other znode, it can call this API repeatedly.
Because we only set one node by this API, the result will be straight,
success or failure. We need not take care of the half-success result.

How do ur guys think about this API?

Thanks~

On Fri, Dec 10, 2010 at 11:57 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Yes.  I can imagine that being able to write several variables, specifying
> the version of each would
> be useful.  But not yet supported.
>
> On Fri, Dec 10, 2010 at 12:01 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>
> > What's more, I think this kind of conditional write support is simpler
> than
> > multiple transactions. Multiple transactions can be built with this kind
> of
> > support. The link is broken?
> >
>

--
With Regards!

Ye, Qian

If this work were to be done, I would much rather prefer the ability
to set multiple values at the same time, as this enables a whole host
of possibilities including certain transactional updating capabilities
that aren't possible now (far too easy to get in an in-consistent
state if you fail partway through updating multiple nodes).
-Dave Wright

On Wed, Dec 15, 2010 at 2:21 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
> I have read the mails on
> http://www.mail-archive.com/[EMAIL PROTECTED]/msg08315.html,
> and here is some further thinking about this issue.
>
> The interface like
>
> zoo_multi_test_and_set(List<int> versions, List<string> znodes,
> List<byte[]> data）
>
> can solve the problem I mentioned before, and some relavant issues, like
> hard for programmers to use, as mentioned in mail-archive, should be paid
> attention to. I think we can move small step first, that is, provide
> interface like
>
> zoo_multi_test_and_set(List<int> versions, List<string> znodes, byte[]
> data, string znode）
>
>
> The API test versions of several different znodes before set one znode, and
> if the client want to set other znode, it can call this API repeatedly.
> Because we only set one node by this API, the result will be straight,
> success or failure. We need not take care of the half-success result.
>
> How do ur guys think about this API?
>
> Thanks~
>
> On Fri, Dec 10, 2010 at 11:57 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>
>> Yes.  I can imagine that being able to write several variables, specifying
>> the version of each would
>> be useful.  But not yet supported.
>>
>> On Fri, Dec 10, 2010 at 12:01 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>>
>> > What's more, I think this kind of conditional write support is simpler
>> than
>> > multiple transactions. Multiple transactions can be built with this kind
>> of
>> > support. The link is broken?
>> >
>>
>
>
>
> --
> With Regards!
>
> Ye, Qian
>

Well, I would just call the first method set.

And I think that the second method is no easier to implement and probably a
bit less useful.

The idea that the second might be almost as useful as the first is
interesting however.  It probably
means that we should allow some of the data elements to be null or something
to allow for testing
versions but not setting data.

On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> wrote:

> zoo_multi_test_and_set(List<int> versions, List<string> znodes,
> List<byte[]> data）
>
> can solve the problem I mentioned before, and some relavant issues, like
> hard for programmers to use, as mentioned in mail-archive, should be paid
> attention to. I think we can move small step first, that is, provide
> interface like
>
> zoo_multi_test_and_set(List<int> versions, List<string> znodes, byte[]
> data, string znode）
>
>
> The API test versions of several different znodes before set one znode, and
> if the client want to set other znode, it can call this API repeatedly.
> Because we only set one node by this API, the result will be straight,
> success or failure. We need not take care of the half-success result.
>
> How do ur guys think about this API?
>

I think the second is easier to implement because it only updates data on
one node, and need not handle rollback in the case of some update failure
after some update success.

Well, I agree that the API
zoo_multi_test_and_set(List<int> versions, List<string> znodes_test,
List<byte[]> data, List<string> znodes_set);

is more powerful. If we can decide to do this, I think we could start to
discuss some implement details.
On Wed, Dec 15, 2010 at 11:50 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Well, I would just call the first method set.
>
> And I think that the second method is no easier to implement and probably a
> bit less useful.
>
> The idea that the second might be almost as useful as the first is
> interesting however.  It probably
> means that we should allow some of the data elements to be null or
> something
> to allow for testing
> versions but not setting data.
>
> On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> wrote:
>
> > zoo_multi_test_and_set(List<int> versions, List<string> znodes,
> > List<byte[]> data）
> >
> > can solve the problem I mentioned before, and some relavant issues, like
> > hard for programmers to use, as mentioned in mail-archive, should be paid
> > attention to. I think we can move small step first, that is, provide
> > interface like
> >
> > zoo_multi_test_and_set(List<int> versions, List<string> znodes, byte[]
> > data, string znode）
> >
> >
> > The API test versions of several different znodes before set one znode,
> and
> > if the client want to set other znode, it can call this API repeatedly.
> > Because we only set one node by this API, the result will be straight,
> > success or failure. We need not take care of the half-success result.
> >
> > How do ur guys think about this API?
> >
>

--
With Regards!

Ye, Qian

The way that zookeeper works internally probably makes these two options
about equal difficulty.  The master has to examine the queue augmented by
the versions in the multi-update in either case.  The only difference is
whether after qualifying the update does it copy multiple one unconditional
update or several.

On Wed, Dec 15, 2010 at 7:20 PM, Qian Ye <[EMAIL PROTECTED]> wrote:

> I think the second is easier to implement because it only updates data on
> one node, and need not handle rollback in the case of some update failure
> after some update success.
>