|
Dave Wright
2010-12-16, 16:16
Ted Dunning
2010-12-16, 18:01
Dave Wright
2010-12-16, 18:06
Jared Cantwell
2010-12-16, 18:39
Henry Robinson
2010-12-16, 19:04
Ted Dunning
2010-12-16, 19:21
Ted Dunning
2010-12-16, 19:22
Ted Dunning
2010-12-16, 19:23
Ted Dunning
2010-12-16, 19:25
Qian Ye
2010-12-21, 02:56
Ted Dunning
2010-12-21, 03:22
Benjamin Reed
2010-12-21, 05:24
Benjamin Reed
2010-12-21, 05:24
Ted Dunning
2010-12-21, 06:08
Benjamin Reed
2010-12-21, 06:57
Jared Cantwell
2010-12-21, 21:16
Henry Robinson
2010-12-21, 21:30
Ted Dunning
2010-12-21, 21:48
Jared Cantwell
2010-12-22, 01:44
Ted Dunning
2010-12-22, 02:06
Henry Robinson
2010-12-22, 09:00
Ted Dunning
2010-12-22, 18:29
|
-
Re: need for more conditional write supportDave Wright 2010-12-16, 16:16
My recommendation would actually be a combination of the two which
offers the most flexibility: zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, List<string> znodesToSet, List<byte[]> data) ...this specifies a list of nodes & versions to check, and if the versions match, a list of nodes to set and the associated data. This allows multiple scenarios, including setting nodes other than the ones you are version checking, setting more nodes than you version check, checking more nodes than you set, etc. I don't think the implementation would be any harder than either of the others. -Dave On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning <[EMAIL PROTECTED]> wrote: > Well, I would just call the first method set. > > And I think that the second method is no easier to implement and probably a > bit less useful. > > The idea that the second might be almost as useful as the first is > interesting however. It probably > means that we should allow some of the data elements to be null or something > to allow for testing > versions but not setting data. > > On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> wrote: > >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, >> List<byte[]> data) >> >> can solve the problem I mentioned before, and some relavant issues, like >> hard for programmers to use, as mentioned in mail-archive, should be paid >> attention to. I think we can move small step first, that is, provide >> interface like >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, byte[] >> data, string znode) >> >> >> The API test versions of several different znodes before set one znode, and >> if the client want to set other znode, it can call this API repeatedly. >> Because we only set one node by this API, the result will be straight, >> success or failure. We need not take care of the half-success result. >> >> How do ur guys think about this API? >> >
-
Re: need for more conditional write supportTed Dunning 2010-12-16, 18:01
Yes. This is isomorphic to my suggestion to allow null data. We should
toss around many options to figure out which is the most congenial idiom. Yours is nice since it has two sets of parallel lists. In java with optional arguments it would be possible to use a builder style with optional arguments: zk.testVersions(node1, version1, node2, version2, ...) .updateData(node1, data1, node3, data3, ...) I would tend to make it part of the contract that the nodes in the second part be a subset of of the nodes in the first part. The first method would create an object packaging up the first set of args and the second method would do the work. Of course, this is just syntactic sugar for the more list oriented version. On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright <[EMAIL PROTECTED]> wrote: > My recommendation would actually be a combination of the two which > offers the most flexibility: > > zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, > List<string> znodesToSet, List<byte[]> data) > > ...this specifies a list of nodes & versions to check, and if the > versions match, a list of nodes to set and the associated data. > This allows multiple scenarios, including setting nodes other than the > ones you are version checking, setting more nodes than you version > check, checking more nodes than you set, etc. > I don't think the implementation would be any harder than either of the > others. > > -Dave > > > On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning <[EMAIL PROTECTED]> > wrote: > > Well, I would just call the first method set. > > > > And I think that the second method is no easier to implement and probably > a > > bit less useful. > > > > The idea that the second might be almost as useful as the first is > > interesting however. It probably > > means that we should allow some of the data elements to be null or > something > > to allow for testing > > versions but not setting data. > > > > On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> wrote: > > > >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, > >> List<byte[]> data) > >> > >> can solve the problem I mentioned before, and some relavant issues, like > >> hard for programmers to use, as mentioned in mail-archive, should be > paid > >> attention to. I think we can move small step first, that is, provide > >> interface like > >> > >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, byte[] > >> data, string znode) > >> > >> > >> The API test versions of several different znodes before set one znode, > and > >> if the client want to set other znode, it can call this API repeatedly. > >> Because we only set one node by this API, the result will be straight, > >> success or failure. We need not take care of the half-success result. > >> > >> How do ur guys think about this API? > >> > > >
-
Re: need for more conditional write supportDave Wright 2010-12-16, 18:06
I'm not sure why (other than your syntax) you would require the second
list (to update) to be a subset of the first (to test). There are plenty of situations where you may want to update one node based on the value of another (and test that the value hasn't changed before updating) but don't really care about the second node, and it would just be extra overhead to check it's current value. In fact, I think that was the OP's situation. -Dave On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning <[EMAIL PROTECTED]> wrote: > Yes. This is isomorphic to my suggestion to allow null data. We should > toss around many options to figure out which is the most congenial idiom. > Yours is nice since it has two sets of parallel lists. > > In java with optional arguments it would be possible to use a builder style > with optional arguments: > > zk.testVersions(node1, version1, node2, version2, ...) > �� .updateData(node1, data1, node3, data3, ...) > > I would tend to make it part of the contract that the nodes in the second > part be a subset of of the nodes in the first part. The first method would > create an object packaging up the first set of args and the second method > would do the work. Of course, this is just syntactic sugar for the more > list oriented version. > > On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright <[EMAIL PROTECTED]> wrote: > >> My recommendation would actually be a combination of the two which >> offers the most flexibility: >> >> zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, >> List<string> znodesToSet, List<byte[]> data) >> >> ...this specifies a list of nodes & versions to check, and if the >> versions match, a list of nodes to set and the associated data. >> This allows multiple scenarios, including setting nodes other than the >> ones you are version checking, setting more nodes than you version >> check, checking more nodes than you set, etc. >> I don't think the implementation would be any harder than either of the >> others. >> >> -Dave >> >> >> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning <[EMAIL PROTECTED]> >> wrote: >> > Well, I would just call the first method set. >> > >> > And I think that the second method is no easier to implement and probably >> a >> > bit less useful. >> > >> > The idea that the second might be almost as useful as the first is >> > interesting however. It probably >> > means that we should allow some of the data elements to be null or >> something >> > to allow for testing >> > versions but not setting data. >> > >> > On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> wrote: >> > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, >> >> List<byte[]> data) >> >> >> >> can solve the problem I mentioned before, and some relavant issues, like >> >> hard for programmers to use, as mentioned in mail-archive, should be >> paid >> >> attention to. I think we can move small step first, that is, provide >> >> interface like >> >> >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, byte[] >> >> data, string znode) >> >> >> >> >> >> The API test versions of several different znodes before set one znode, >> and >> >> if the client want to set other znode, it can call this API repeatedly. >> >> Because we only set one node by this API, the result will be straight, >> >> success or failure. We need not take care of the half-success result. >> >> >> >> How do ur guys think about this API? >> >> >> > >> >
-
Re: need for more conditional write supportJared Cantwell 2010-12-16, 18:39
I think that syntactic sugar can be very limiting. What if you have X
children you would like to update, but don't know X until runtime? I like the idea of lists that don't have to be subsets of each other, giving more flexibility. I also think it would be interesting to discuss what additional recipes could be developed with this api. ~Jared On Thu, Dec 16, 2010 at 1:06 PM, Dave Wright <[EMAIL PROTECTED]> wrote: > I'm not sure why (other than your syntax) you would require the second > list (to update) to be a subset of the first (to test). There are > plenty of situations where you may want to update one node based on > the value of another (and test that the value hasn't changed before > updating) but don't really care about the second node, and it would > just be extra overhead to check it's current value. In fact, I think > that was the OP's situation. > > -Dave > > On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning <[EMAIL PROTECTED]> > wrote: > > Yes. This is isomorphic to my suggestion to allow null data. We should > > toss around many options to figure out which is the most congenial idiom. > > Yours is nice since it has two sets of parallel lists. > > > > In java with optional arguments it would be possible to use a builder > style > > with optional arguments: > > > > zk.testVersions(node1, version1, node2, version2, ...) > > .updateData(node1, data1, node3, data3, ...) > > > > I would tend to make it part of the contract that the nodes in the second > > part be a subset of of the nodes in the first part. The first method > would > > create an object packaging up the first set of args and the second method > > would do the work. Of course, this is just syntactic sugar for the more > > list oriented version. > > > > On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright <[EMAIL PROTECTED]> wrote: > > > >> My recommendation would actually be a combination of the two which > >> offers the most flexibility: > >> > >> zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, > >> List<string> znodesToSet, List<byte[]> data) > >> > >> ...this specifies a list of nodes & versions to check, and if the > >> versions match, a list of nodes to set and the associated data. > >> This allows multiple scenarios, including setting nodes other than the > >> ones you are version checking, setting more nodes than you version > >> check, checking more nodes than you set, etc. > >> I don't think the implementation would be any harder than either of the > >> others. > >> > >> -Dave > >> > >> > >> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning <[EMAIL PROTECTED]> > >> wrote: > >> > Well, I would just call the first method set. > >> > > >> > And I think that the second method is no easier to implement and > probably > >> a > >> > bit less useful. > >> > > >> > The idea that the second might be almost as useful as the first is > >> > interesting however. It probably > >> > means that we should allow some of the data elements to be null or > >> something > >> > to allow for testing > >> > versions but not setting data. > >> > > >> > On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> > wrote: > >> > > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, > >> >> List<byte[]> data) > >> >> > >> >> can solve the problem I mentioned before, and some relavant issues, > like > >> >> hard for programmers to use, as mentioned in mail-archive, should be > >> paid > >> >> attention to. I think we can move small step first, that is, provide > >> >> interface like > >> >> > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, > byte[] > >> >> data, string znode) > >> >> > >> >> > >> >> The API test versions of several different znodes before set one > znode, > >> and > >> >> if the client want to set other znode, it can call this API > repeatedly. > >> >> Because we only set one node by this API, the result will be > straight, > >> >> success or failure. We need not take care of the half-success result.
-
Re: need for more conditional write supportHenry Robinson 2010-12-16, 19:04
I expect any of these proposals will be relatively simple to implement. My
understanding is that ZK serialises *all* accesses to the data tree, so there's no need to worry about acquiring individual locks for each znode. This should be a cautionary note on performance, however: as there is no parallelism in the execution of updates (although there is plenty in the serialisation process) we should build a mechanism to constrain how much work this operation can perform, otherwise there's a danger of hurting throughput for all clients of a cluster. ZK could do with namespaces, or a cleverer locking mechanism like traditional databases use, to mitigate this issue but that's a much larger undertaking. Figuring out a good API for the ensemble to implement can be slightly decoupled from the API that a client application sees. Therefore I prefer the list parameters API, which can be wrapped in a builder API in Java if that makes sense (these kinds of API are less natural in C, for example). cheers, Henry On 16 December 2010 10:39, Jared Cantwell <[EMAIL PROTECTED]> wrote: > I think that syntactic sugar can be very limiting. What if you have X > children you would like to update, but don't know X until runtime? I like > the idea of lists that don't have to be subsets of each other, giving more > flexibility. I also think it would be interesting to discuss what > additional recipes could be developed with this api. > > ~Jared > > On Thu, Dec 16, 2010 at 1:06 PM, Dave Wright <[EMAIL PROTECTED]> wrote: > > > I'm not sure why (other than your syntax) you would require the second > > list (to update) to be a subset of the first (to test). There are > > plenty of situations where you may want to update one node based on > > the value of another (and test that the value hasn't changed before > > updating) but don't really care about the second node, and it would > > just be extra overhead to check it's current value. In fact, I think > > that was the OP's situation. > > > > -Dave > > > > On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning <[EMAIL PROTECTED]> > > wrote: > > > Yes. This is isomorphic to my suggestion to allow null data. We > should > > > toss around many options to figure out which is the most congenial > idiom. > > > Yours is nice since it has two sets of parallel lists. > > > > > > In java with optional arguments it would be possible to use a builder > > style > > > with optional arguments: > > > > > > zk.testVersions(node1, version1, node2, version2, ...) > > > .updateData(node1, data1, node3, data3, ...) > > > > > > I would tend to make it part of the contract that the nodes in the > second > > > part be a subset of of the nodes in the first part. The first method > > would > > > create an object packaging up the first set of args and the second > method > > > would do the work. Of course, this is just syntactic sugar for the > more > > > list oriented version. > > > > > > On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright <[EMAIL PROTECTED]> > wrote: > > > > > >> My recommendation would actually be a combination of the two which > > >> offers the most flexibility: > > >> > > >> zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, > > >> List<string> znodesToSet, List<byte[]> data) > > >> > > >> ...this specifies a list of nodes & versions to check, and if the > > >> versions match, a list of nodes to set and the associated data. > > >> This allows multiple scenarios, including setting nodes other than the > > >> ones you are version checking, setting more nodes than you version > > >> check, checking more nodes than you set, etc. > > >> I don't think the implementation would be any harder than either of > the > > >> others. > > >> > > >> -Dave > > >> > > >> > > >> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning <[EMAIL PROTECTED]> > > >> wrote: > > >> > Well, I would just call the first method set. > > >> > > > >> > And I think that the second method is no easier to implement and Henry Robinson Software Engineer Cloudera 415-994-6679
-
Re: need for more conditional write supportTed Dunning 2010-12-16, 19:21
My thought is that this should be handled by limiting the *total* size of
all updates in one call to be limited the same way that *single* update sizes are limited now. On Thu, Dec 16, 2010 at 11:04 AM, Henry Robinson <[EMAIL PROTECTED]> wrote: > This should be a cautionary note on performance, however: as there is no > parallelism in the execution of updates (although there is plenty in the > serialisation process) we should build a mechanism to constrain how much > work this operation can perform, otherwise there's a danger of hurting > throughput for all clients of a cluster. >
-
Re: need for more conditional write supportTed Dunning 2010-12-16, 19:22
In that case, you use the lower level unsugared version.
On Thu, Dec 16, 2010 at 10:39 AM, Jared Cantwell <[EMAIL PROTECTED]>wrote: > I think that syntactic sugar can be very limiting. What if you have X > children you would like to update, but don't know X until runtime? >
-
Re: need for more conditional write supportTed Dunning 2010-12-16, 19:23
I oscillate on this issue every 10 minutes. I expect that non subset
updates will be an error 80% of the time, but there is a very valid 20% of the time where it would be lovely to have. On Thu, Dec 16, 2010 at 10:39 AM, Jared Cantwell <[EMAIL PROTECTED]>wrote: > I like > the idea of lists that don't have to be subsets of each other, giving more > flexibility. I also think it would be interesting to discuss what > additional recipes could be developed with this api. >
-
Re: need for more conditional write supportTed Dunning 2010-12-16, 19:25
One alternative is to simply specify -1 in the version list to avoid the
version check for that one item. That would allow the subset constraint to be retained as a valid semantic check for most situations and would allow a very explicit way to describe when you want to violate that constraint. On Thu, Dec 16, 2010 at 10:06 AM, Dave Wright <[EMAIL PROTECTED]> wrote: > I'm not sure why (other than your syntax) you would require the second > list (to update) to be a subset of the first (to test). There are > plenty of situations where you may want to update one node based on > the value of another (and test that the value hasn't changed before > updating) but don't really care about the second node, and it would > just be extra overhead to check it's current value. In fact, I think > that was the OP's situation. > > -Dave > > On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning <[EMAIL PROTECTED]> > wrote: > > Yes. This is isomorphic to my suggestion to allow null data. We should > > toss around many options to figure out which is the most congenial idiom. > > Yours is nice since it has two sets of parallel lists. > > > > In java with optional arguments it would be possible to use a builder > style > > with optional arguments: > > > > zk.testVersions(node1, version1, node2, version2, ...) > > .updateData(node1, data1, node3, data3, ...) > > > > I would tend to make it part of the contract that the nodes in the second > > part be a subset of of the nodes in the first part. The first method > would > > create an object packaging up the first set of args and the second method > > would do the work. Of course, this is just syntactic sugar for the more > > list oriented version. > > > > On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright <[EMAIL PROTECTED]> wrote: > > > >> My recommendation would actually be a combination of the two which > >> offers the most flexibility: > >> > >> zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, > >> List<string> znodesToSet, List<byte[]> data) > >> > >> ...this specifies a list of nodes & versions to check, and if the > >> versions match, a list of nodes to set and the associated data. > >> This allows multiple scenarios, including setting nodes other than the > >> ones you are version checking, setting more nodes than you version > >> check, checking more nodes than you set, etc. > >> I don't think the implementation would be any harder than either of the > >> others. > >> > >> -Dave > >> > >> > >> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning <[EMAIL PROTECTED]> > >> wrote: > >> > Well, I would just call the first method set. > >> > > >> > And I think that the second method is no easier to implement and > probably > >> a > >> > bit less useful. > >> > > >> > The idea that the second might be almost as useful as the first is > >> > interesting however. It probably > >> > means that we should allow some of the data elements to be null or > >> something > >> > to allow for testing > >> > versions but not setting data. > >> > > >> > On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> > wrote: > >> > > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, > >> >> List<byte[]> data) > >> >> > >> >> can solve the problem I mentioned before, and some relavant issues, > like > >> >> hard for programmers to use, as mentioned in mail-archive, should be > >> paid > >> >> attention to. I think we can move small step first, that is, provide > >> >> interface like > >> >> > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, > byte[] > >> >> data, string znode) > >> >> > >> >> > >> >> The API test versions of several different znodes before set one > znode, > >> and > >> >> if the client want to set other znode, it can call this API > repeatedly. > >> >> Because we only set one node by this API, the result will be > straight, > >> >> success or failure. We need not take care of the half-success result. > >> >>
-
Re: need for more conditional write supportQian Ye 2010-12-21, 02:56
Hi all, have we reached any consensus on this issue? What's our next step
about it? I'm looking forward to make use of this kind of feature. thanks~ On Fri, Dec 17, 2010 at 3:25 AM, Ted Dunning <[EMAIL PROTECTED]> wrote: > One alternative is to simply specify -1 in the version list to avoid the > version check for that one item. That would allow > the subset constraint to be retained as a valid semantic check for most > situations and would allow > a very explicit way to describe when you want to violate that constraint. > > On Thu, Dec 16, 2010 at 10:06 AM, Dave Wright <[EMAIL PROTECTED]> wrote: > > > I'm not sure why (other than your syntax) you would require the second > > list (to update) to be a subset of the first (to test). There are > > plenty of situations where you may want to update one node based on > > the value of another (and test that the value hasn't changed before > > updating) but don't really care about the second node, and it would > > just be extra overhead to check it's current value. In fact, I think > > that was the OP's situation. > > > > -Dave > > > > On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning <[EMAIL PROTECTED]> > > wrote: > > > Yes. This is isomorphic to my suggestion to allow null data. We > should > > > toss around many options to figure out which is the most congenial > idiom. > > > Yours is nice since it has two sets of parallel lists. > > > > > > In java with optional arguments it would be possible to use a builder > > style > > > with optional arguments: > > > > > > zk.testVersions(node1, version1, node2, version2, ...) > > > .updateData(node1, data1, node3, data3, ...) > > > > > > I would tend to make it part of the contract that the nodes in the > second > > > part be a subset of of the nodes in the first part. The first method > > would > > > create an object packaging up the first set of args and the second > method > > > would do the work. Of course, this is just syntactic sugar for the > more > > > list oriented version. > > > > > > On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright <[EMAIL PROTECTED]> > wrote: > > > > > >> My recommendation would actually be a combination of the two which > > >> offers the most flexibility: > > >> > > >> zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, > > >> List<string> znodesToSet, List<byte[]> data) > > >> > > >> ...this specifies a list of nodes & versions to check, and if the > > >> versions match, a list of nodes to set and the associated data. > > >> This allows multiple scenarios, including setting nodes other than the > > >> ones you are version checking, setting more nodes than you version > > >> check, checking more nodes than you set, etc. > > >> I don't think the implementation would be any harder than either of > the > > >> others. > > >> > > >> -Dave > > >> > > >> > > >> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning <[EMAIL PROTECTED]> > > >> wrote: > > >> > Well, I would just call the first method set. > > >> > > > >> > And I think that the second method is no easier to implement and > > probably > > >> a > > >> > bit less useful. > > >> > > > >> > The idea that the second might be almost as useful as the first is > > >> > interesting however. It probably > > >> > means that we should allow some of the data elements to be null or > > >> something > > >> > to allow for testing > > >> > versions but not setting data. > > >> > > > >> > On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> > > wrote: > > >> > > > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, > > >> >> List<byte[]> data) > > >> >> > > >> >> can solve the problem I mentioned before, and some relavant issues, > > like > > >> >> hard for programmers to use, as mentioned in mail-archive, should > be > > >> paid > > >> >> attention to. I think we can move small step first, that is, > provide > > >> >> interface like > > >> >> > > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, With Regards! Ye, Qian
-
Re: need for more conditional write supportTed Dunning 2010-12-21, 03:22
Ben,
Can you point me to the place where the queue is inspected to convert updates into idempotent writes? On Mon, Dec 20, 2010 at 6:56 PM, Qian Ye <[EMAIL PROTECTED]> wrote: > Hi all, have we reached any consensus on this issue? What's our next step > about it? > I'm looking forward to make use of this kind of feature. > > thanks~ > > On Fri, Dec 17, 2010 at 3:25 AM, Ted Dunning <[EMAIL PROTECTED]> > wrote: > > > One alternative is to simply specify -1 in the version list to avoid the > > version check for that one item. That would allow > > the subset constraint to be retained as a valid semantic check for most > > situations and would allow > > a very explicit way to describe when you want to violate that constraint. > > > > On Thu, Dec 16, 2010 at 10:06 AM, Dave Wright <[EMAIL PROTECTED]> wrote: > > > > > I'm not sure why (other than your syntax) you would require the second > > > list (to update) to be a subset of the first (to test). There are > > > plenty of situations where you may want to update one node based on > > > the value of another (and test that the value hasn't changed before > > > updating) but don't really care about the second node, and it would > > > just be extra overhead to check it's current value. In fact, I think > > > that was the OP's situation. > > > > > > -Dave > > > > > > On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning <[EMAIL PROTECTED]> > > > wrote: > > > > Yes. This is isomorphic to my suggestion to allow null data. We > > should > > > > toss around many options to figure out which is the most congenial > > idiom. > > > > Yours is nice since it has two sets of parallel lists. > > > > > > > > In java with optional arguments it would be possible to use a builder > > > style > > > > with optional arguments: > > > > > > > > zk.testVersions(node1, version1, node2, version2, ...) > > > > .updateData(node1, data1, node3, data3, ...) > > > > > > > > I would tend to make it part of the contract that the nodes in the > > second > > > > part be a subset of of the nodes in the first part. The first method > > > would > > > > create an object packaging up the first set of args and the second > > method > > > > would do the work. Of course, this is just syntactic sugar for the > > more > > > > list oriented version. > > > > > > > > On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright <[EMAIL PROTECTED]> > > wrote: > > > > > > > >> My recommendation would actually be a combination of the two which > > > >> offers the most flexibility: > > > >> > > > >> zoo_multi_test_and_set(List<string> znodesToTest, List<int> > versions, > > > >> List<string> znodesToSet, List<byte[]> data) > > > >> > > > >> ...this specifies a list of nodes & versions to check, and if the > > > >> versions match, a list of nodes to set and the associated data. > > > >> This allows multiple scenarios, including setting nodes other than > the > > > >> ones you are version checking, setting more nodes than you version > > > >> check, checking more nodes than you set, etc. > > > >> I don't think the implementation would be any harder than either of > > the > > > >> others. > > > >> > > > >> -Dave > > > >> > > > >> > > > >> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning < > [EMAIL PROTECTED]> > > > >> wrote: > > > >> > Well, I would just call the first method set. > > > >> > > > > >> > And I think that the second method is no easier to implement and > > > probably > > > >> a > > > >> > bit less useful. > > > >> > > > > >> > The idea that the second might be almost as useful as the first is > > > >> > interesting however. It probably > > > >> > means that we should allow some of the data elements to be null or > > > >> something > > > >> > to allow for testing > > > >> > versions but not setting data. > > > >> > > > > >> > On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye <[EMAIL PROTECTED]> > > > wrote: > > > >> > > > > >> >> zoo_multi_test_and_set(List<int> versions, List<string> znodes, > > > >>
-
Re: need for more conditional write supportBenjamin Reed 2010-12-21, 05:24
It all happens in PrepRequestProcessor.
ben On 12/20/2010 07:22 PM, Ted Dunning wrote: > Ben, > > Can you point me to the place where the queue is inspected to convert > updates into idempotent writes? > > On Mon, Dec 20, 2010 at 6:56 PM, Qian Ye<[EMAIL PROTECTED]> wrote: > >> Hi all, have we reached any consensus on this issue? What's our next step >> about it? >> I'm looking forward to make use of this kind of feature. >> >> thanks~ >> >> On Fri, Dec 17, 2010 at 3:25 AM, Ted Dunning<[EMAIL PROTECTED]> >> wrote: >> >>> One alternative is to simply specify -1 in the version list to avoid the >>> version check for that one item. That would allow >>> the subset constraint to be retained as a valid semantic check for most >>> situations and would allow >>> a very explicit way to describe when you want to violate that constraint. >>> >>> On Thu, Dec 16, 2010 at 10:06 AM, Dave Wright<[EMAIL PROTECTED]> wrote: >>> >>>> I'm not sure why (other than your syntax) you would require the second >>>> list (to update) to be a subset of the first (to test). There are >>>> plenty of situations where you may want to update one node based on >>>> the value of another (and test that the value hasn't changed before >>>> updating) but don't really care about the second node, and it would >>>> just be extra overhead to check it's current value. In fact, I think >>>> that was the OP's situation. >>>> >>>> -Dave >>>> >>>> On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning<[EMAIL PROTECTED]> >>>> wrote: >>>>> Yes. This is isomorphic to my suggestion to allow null data. We >>> should >>>>> toss around many options to figure out which is the most congenial >>> idiom. >>>>> Yours is nice since it has two sets of parallel lists. >>>>> >>>>> In java with optional arguments it would be possible to use a builder >>>> style >>>>> with optional arguments: >>>>> >>>>> zk.testVersions(node1, version1, node2, version2, ...) >>>>> .updateData(node1, data1, node3, data3, ...) >>>>> >>>>> I would tend to make it part of the contract that the nodes in the >>> second >>>>> part be a subset of of the nodes in the first part. The first method >>>> would >>>>> create an object packaging up the first set of args and the second >>> method >>>>> would do the work. Of course, this is just syntactic sugar for the >>> more >>>>> list oriented version. >>>>> >>>>> On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright<[EMAIL PROTECTED]> >>> wrote: >>>>>> My recommendation would actually be a combination of the two which >>>>>> offers the most flexibility: >>>>>> >>>>>> zoo_multi_test_and_set(List<string> znodesToTest, List<int> >> versions, >>>>>> List<string> znodesToSet, List<byte[]> data) >>>>>> >>>>>> ...this specifies a list of nodes& versions to check, and if the >>>>>> versions match, a list of nodes to set and the associated data. >>>>>> This allows multiple scenarios, including setting nodes other than >> the >>>>>> ones you are version checking, setting more nodes than you version >>>>>> check, checking more nodes than you set, etc. >>>>>> I don't think the implementation would be any harder than either of >>> the >>>>>> others. >>>>>> >>>>>> -Dave >>>>>> >>>>>> >>>>>> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning< >> [EMAIL PROTECTED]> >>>>>> wrote: >>>>>>> Well, I would just call the first method set. >>>>>>> >>>>>>> And I think that the second method is no easier to implement and >>>> probably >>>>>> a >>>>>>> bit less useful. >>>>>>> >>>>>>> The idea that the second might be almost as useful as the first is >>>>>>> interesting however. It probably >>>>>>> means that we should allow some of the data elements to be null or >>>>>> something >>>>>>> to allow for testing >>>>>>> versions but not setting data. >>>>>>> >>>>>>> On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye<[EMAIL PROTECTED]> >>>> wrote: >>>>>>>> zoo_multi_test_and_set(List<int> versions, List<string> znodes, >>>>>>>> List<byte[]> data) >>>>>>>> >>>>>>
-
Re: need for more conditional write supportBenjamin Reed 2010-12-21, 05:24
are you guys going to put a limit on the size of the updates? can
someone do an update over 50 znodes where data value is 500K, for example? if there is a failure during the update, is it okay for just a subset of the znodes to be updated? ben On 12/20/2010 06:56 PM, Qian Ye wrote: > Hi all, have we reached any consensus on this issue? What's our next step > about it? > I'm looking forward to make use of this kind of feature. > > thanks~ > > On Fri, Dec 17, 2010 at 3:25 AM, Ted Dunning<[EMAIL PROTECTED]> wrote: > >> One alternative is to simply specify -1 in the version list to avoid the >> version check for that one item. That would allow >> the subset constraint to be retained as a valid semantic check for most >> situations and would allow >> a very explicit way to describe when you want to violate that constraint. >> >> On Thu, Dec 16, 2010 at 10:06 AM, Dave Wright<[EMAIL PROTECTED]> wrote: >> >>> I'm not sure why (other than your syntax) you would require the second >>> list (to update) to be a subset of the first (to test). There are >>> plenty of situations where you may want to update one node based on >>> the value of another (and test that the value hasn't changed before >>> updating) but don't really care about the second node, and it would >>> just be extra overhead to check it's current value. In fact, I think >>> that was the OP's situation. >>> >>> -Dave >>> >>> On Thu, Dec 16, 2010 at 1:01 PM, Ted Dunning<[EMAIL PROTECTED]> >>> wrote: >>>> Yes. This is isomorphic to my suggestion to allow null data. We >> should >>>> toss around many options to figure out which is the most congenial >> idiom. >>>> Yours is nice since it has two sets of parallel lists. >>>> >>>> In java with optional arguments it would be possible to use a builder >>> style >>>> with optional arguments: >>>> >>>> zk.testVersions(node1, version1, node2, version2, ...) >>>> .updateData(node1, data1, node3, data3, ...) >>>> >>>> I would tend to make it part of the contract that the nodes in the >> second >>>> part be a subset of of the nodes in the first part. The first method >>> would >>>> create an object packaging up the first set of args and the second >> method >>>> would do the work. Of course, this is just syntactic sugar for the >> more >>>> list oriented version. >>>> >>>> On Thu, Dec 16, 2010 at 8:16 AM, Dave Wright<[EMAIL PROTECTED]> >> wrote: >>>>> My recommendation would actually be a combination of the two which >>>>> offers the most flexibility: >>>>> >>>>> zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, >>>>> List<string> znodesToSet, List<byte[]> data) >>>>> >>>>> ...this specifies a list of nodes& versions to check, and if the >>>>> versions match, a list of nodes to set and the associated data. >>>>> This allows multiple scenarios, including setting nodes other than the >>>>> ones you are version checking, setting more nodes than you version >>>>> check, checking more nodes than you set, etc. >>>>> I don't think the implementation would be any harder than either of >> the >>>>> others. >>>>> >>>>> -Dave >>>>> >>>>> >>>>> On Wed, Dec 15, 2010 at 10:50 AM, Ted Dunning<[EMAIL PROTECTED]> >>>>> wrote: >>>>>> Well, I would just call the first method set. >>>>>> >>>>>> And I think that the second method is no easier to implement and >>> probably >>>>> a >>>>>> bit less useful. >>>>>> >>>>>> The idea that the second might be almost as useful as the first is >>>>>> interesting however. It probably >>>>>> means that we should allow some of the data elements to be null or >>>>> something >>>>>> to allow for testing >>>>>> versions but not setting data. >>>>>> >>>>>> On Tue, Dec 14, 2010 at 11:21 PM, Qian Ye<[EMAIL PROTECTED]> >>> wrote: >>>>>>> zoo_multi_test_and_set(List<int> versions, List<string> znodes, >>>>>>> List<byte[]> data) >>>>>>> >>>>>>> can solve the problem I mentioned before, and some relavant issues, >>> like >>>>>>> hard for programmers to use, as mentioned in mail-archive, should
-
Re: need for more conditional write supportTed Dunning 2010-12-21, 06:08
On Mon, Dec 20, 2010 at 9:24 PM, Benjamin Reed <[EMAIL PROTECTED]> wrote:
> are you guys going to put a limit on the size of the updates? can someone > do an update over 50 znodes where data value is 500K, for example? > Yes. My plan is to put a limit on the aggregate size of all of the updates that is equal to the limit that gets put on a single update normally. > if there is a failure during the update, is it okay for just a subset of > the znodes to be updated? > That would be an unpleasant alternative. My thought was to convert all of the updates to idempotent form and add them all to the queue or fail all the updates. My hope was that there would be some way to mark the batch in the queue so that they stay together when commits are pushed out to the cluster. It might be necessary to flush the queue before inserting the batched updates. Presumably something like this needs to be done now (if queue + current transaction is too large, flush queue first). Are there failure modes that would leave part of the queue committed and part not?
-
Re: need for more conditional write supportBenjamin Reed 2010-12-21, 06:57
keeping the aggregate size to the normal max i think helps things a lot.
we don't have to worry about a big update slowing everything down. to implement this we probably need to add a new request and a new transaction. then you will get the atomic update property that you are looking for and you will not need to worry about special queue management. ben On 12/20/2010 10:08 PM, Ted Dunning wrote: > On Mon, Dec 20, 2010 at 9:24 PM, Benjamin Reed<[EMAIL PROTECTED]> wrote: > >> are you guys going to put a limit on the size of the updates? can someone >> do an update over 50 znodes where data value is 500K, for example? >> > Yes. My plan is to put a limit on the aggregate size of all of the updates > that is equal to the limit that gets put on a single update normally. > > >> if there is a failure during the update, is it okay for just a subset of >> the znodes to be updated? >> > That would be an unpleasant alternative. > > My thought was to convert all of the updates to idempotent form and add them > all to the queue or fail all the updates. > > My hope was that there would be some way to mark the batch in the queue so > that they stay together when commits are pushed out to the cluster. It > might be necessary to flush the queue before inserting the batched updates. > Presumably something like this needs to be done now (if queue + current > transaction is too large, flush queue first). > > Are there failure modes that would leave part of the queue committed and > part not?
-
Re: need for more conditional write supportJared Cantwell 2010-12-21, 21:16
Based on the discussion, it doesn't look like create/delete actions would be
considered under this model. How difficult would it be to extend the api to allow creation/deletion of nodes? I think the hardest part would be to verify the 'correctness' of the update. Are there other complications? ~Jared On Tue, Dec 21, 2010 at 1:57 AM, Benjamin Reed <[EMAIL PROTECTED]> wrote: > keeping the aggregate size to the normal max i think helps things a lot. we > don't have to worry about a big update slowing everything down. > > to implement this we probably need to add a new request and a new > transaction. then you will get the atomic update property that you are > looking for and you will not need to worry about special queue management. > > ben > > > On 12/20/2010 10:08 PM, Ted Dunning wrote: > >> On Mon, Dec 20, 2010 at 9:24 PM, Benjamin Reed<[EMAIL PROTECTED]> >> wrote: >> >> are you guys going to put a limit on the size of the updates? can someone >>> do an update over 50 znodes where data value is 500K, for example? >>> >>> Yes. My plan is to put a limit on the aggregate size of all of the >> updates >> that is equal to the limit that gets put on a single update normally. >> >> >> if there is a failure during the update, is it okay for just a subset of >>> the znodes to be updated? >>> >>> That would be an unpleasant alternative. >> >> My thought was to convert all of the updates to idempotent form and add >> them >> all to the queue or fail all the updates. >> >> My hope was that there would be some way to mark the batch in the queue so >> that they stay together when commits are pushed out to the cluster. It >> might be necessary to flush the queue before inserting the batched >> updates. >> Presumably something like this needs to be done now (if queue + current >> transaction is too large, flush queue first). >> >> Are there failure modes that would leave part of the queue committed and >> part not? >> > >
-
Re: need for more conditional write supportHenry Robinson 2010-12-21, 21:30
This is a more complicated requirement, and not something that can be done
right now with Zookeeper even in a single operation on a single znode. The only conditional operations are updates. Designing an API to support three different kinds of operation would also be more complicated, and the implementation would be trickier. Not to say that it can't be done, or wouldn't be useful, but I suggest getting the bulk test-and-set API done first: we're near consensus on how the API should look and the implementation will not be too complex. We can then revisit conditional create and delete both for the single operation and the bulk operation cases. Henry On 21 December 2010 13:16, Jared Cantwell <[EMAIL PROTECTED]> wrote: > Based on the discussion, it doesn't look like create/delete actions would > be > considered under this model. How difficult would it be to extend the api > to > allow creation/deletion of nodes? I think the hardest part would be to > verify the 'correctness' of the update. Are there other complications? > > ~Jared > > On Tue, Dec 21, 2010 at 1:57 AM, Benjamin Reed <[EMAIL PROTECTED]> > wrote: > > > keeping the aggregate size to the normal max i think helps things a lot. > we > > don't have to worry about a big update slowing everything down. > > > > to implement this we probably need to add a new request and a new > > transaction. then you will get the atomic update property that you are > > looking for and you will not need to worry about special queue > management. > > > > ben > > > > > > On 12/20/2010 10:08 PM, Ted Dunning wrote: > > > >> On Mon, Dec 20, 2010 at 9:24 PM, Benjamin Reed<[EMAIL PROTECTED]> > >> wrote: > >> > >> are you guys going to put a limit on the size of the updates? can > someone > >>> do an update over 50 znodes where data value is 500K, for example? > >>> > >>> Yes. My plan is to put a limit on the aggregate size of all of the > >> updates > >> that is equal to the limit that gets put on a single update normally. > >> > >> > >> if there is a failure during the update, is it okay for just a subset > of > >>> the znodes to be updated? > >>> > >>> That would be an unpleasant alternative. > >> > >> My thought was to convert all of the updates to idempotent form and add > >> them > >> all to the queue or fail all the updates. > >> > >> My hope was that there would be some way to mark the batch in the queue > so > >> that they stay together when commits are pushed out to the cluster. It > >> might be necessary to flush the queue before inserting the batched > >> updates. > >> Presumably something like this needs to be done now (if queue + current > >> transaction is too large, flush queue first). > >> > >> Are there failure modes that would leave part of the queue committed and > >> part not? > >> > > > > > -- Henry Robinson Software Engineer Cloudera 415-994-6679
-
Re: need for more conditional write supportTed Dunning 2010-12-21, 21:48
On Tue, Dec 21, 2010 at 1:30 PM, Henry Robinson <[EMAIL PROTECTED]> wrote:
> This is a more complicated requirement, and not something that can be done > right now with Zookeeper even in a single operation on a single znode. The > only conditional operations are updates. Actually, creates and deletes are conditional also. Creates are conditional on the file not existing previously and deletes can accept a version number. > Designing an API to support three > different kinds of operation would also be more complicated, and the > implementation would be trickier. > This is definitely true. Conceptually it is the same thing, but practically building a usable API is a bit tricky. Using a builder style would work, though. For example: Zookeeper zk = ...; zk.transaction() .create("/foobar") .update("/pig", data, version) .delete("/dog"", otherVersion) .commit() Another option would be to pass a list of operations which can be one of Create, Update or delete.
-
Re: need for more conditional write supportJared Cantwell 2010-12-22, 01:44
I think this is the interface Ted eluded to, but I wanted to throw it out
there more concretely: zoo_multi_test_and_set(List<string> znodesToTest, List<int> versions, List<string> znodesToModify, List<OperationType> typeOfModifications, List<byte[]> data) You can still keep it a subset by requiring that creations explicitly set the test version to -1. This type of interface would really open up the capability of multi_test_and_set(). I have no idea how much more complicated it would be to implement, but I can imagine each operation type having Test() and Commit() operations. I think that by limiting it to only updates from the start we may do work that will need reversed to make this extension on the interface functional. ~Jared On Tue, Dec 21, 2010 at 4:48 PM, Ted Dunning <[EMAIL PROTECTED]> wrote: > On Tue, Dec 21, 2010 at 1:30 PM, Henry Robinson <[EMAIL PROTECTED]> > wrote: > > > This is a more complicated requirement, and not something that can be > done > > right now with Zookeeper even in a single operation on a single znode. > The > > only conditional operations are updates. > > > Actually, creates and deletes are conditional also. Creates are > conditional > on the file > not existing previously and deletes can accept a version number. > > > > Designing an API to support three > > different kinds of operation would also be more complicated, and the > > implementation would be trickier. > > > > This is definitely true. Conceptually it is the same thing, but > practically > building a usable API is > a bit tricky. Using a builder style would work, though. For example: > > Zookeeper zk = ...; > > > zk.transaction() > .create("/foobar") > .update("/pig", data, version) > .delete("/dog"", otherVersion) > .commit() > > Another option would be to pass a list of operations which can be one of > Create, Update or delete. >
-
Re: need for more conditional write supportTed Dunning 2010-12-22, 02:06
I don't like limiting the operations at the top level. Here is an
alternative: zk.multiOp(List<Operation> operations) Where we have a polymorphic operation: public class Operation { private static String path; public static class Check { public Check(String path, int version) { } } public static class Create { public Create(String path, byte[] data) { } } public static class Update { public Update(String path, byte[] date, int version) { } } public static class Delete { public Delete(String path, int version) { } } } On Tue, Dec 21, 2010 at 5:44 PM, Jared Cantwell <[EMAIL PROTECTED]>wrote: > I think this is the interface Ted eluded to, but I wanted to throw it out > there more concretely: > > zoo_multi_test_and_set(List<string> znodesToTest, List<int> > versions, List<string> znodesToModify, List<OperationType> > typeOfModifications, List<byte[]> data) > > You can still keep it a subset by requiring that creations explicitly set > the test version to -1. This type of interface would really open up the > capability of multi_test_and_set(). I have no idea how much more > complicated it would be to implement, but I can imagine each operation type > having Test() and Commit() operations. I think that by limiting it to only > updates from the start we may do work that will need reversed to make this > extension on the interface functional. > > ~Jared > > On Tue, Dec 21, 2010 at 4:48 PM, Ted Dunning <[EMAIL PROTECTED]> > wrote: > > > On Tue, Dec 21, 2010 at 1:30 PM, Henry Robinson <[EMAIL PROTECTED]> > > wrote: > > > > > This is a more complicated requirement, and not something that can be > > done > > > right now with Zookeeper even in a single operation on a single znode. > > The > > > only conditional operations are updates. > > > > > > Actually, creates and deletes are conditional also. Creates are > > conditional > > on the file > > not existing previously and deletes can accept a version number. > > > > > > > Designing an API to support three > > > different kinds of operation would also be more complicated, and the > > > implementation would be trickier. > > > > > > > This is definitely true. Conceptually it is the same thing, but > > practically > > building a usable API is > > a bit tricky. Using a builder style would work, though. For example: > > > > Zookeeper zk = ...; > > > > > > zk.transaction() > > .create("/foobar") > > .update("/pig", data, version) > > .delete("/dog"", otherVersion) > > .commit() > > > > Another option would be to pass a list of operations which can be one of > > Create, Update or delete. > > >
-
Re: need for more conditional write supportHenry Robinson 2010-12-22, 09:00
On 21 December 2010 13:48, Ted Dunning <[EMAIL PROTECTED]> wrote:
> On Tue, Dec 21, 2010 at 1:30 PM, Henry Robinson <[EMAIL PROTECTED]> > wrote: > > > This is a more complicated requirement, and not something that can be > done > > right now with Zookeeper even in a single operation on a single znode. > The > > only conditional operations are updates. > > > Actually, creates and deletes are conditional also. Creates are > conditional > on the file > not existing previously and deletes can accept a version number. > > Thanks Ted - yes, I misspoke. I was thinking of conditional creates (on inspection of some znode's version) and extrapolated to deletes. > > > Designing an API to support three > > different kinds of operation would also be more complicated, and the > > implementation would be trickier. > > > > This is definitely true. Conceptually it is the same thing, but > practically > building a usable API is > a bit tricky. Using a builder style would work, though. For example: > > Zookeeper zk = ...; > > > zk.transaction() > .create("/foobar") > .update("/pig", data, version) > .delete("/dog"", otherVersion) > .commit() > > Another option would be to pass a list of operations which can be one of > Create, Update or delete. > A builder API is a bit of a departure from the core ZK API as it stands. I'd rather see something like: zk_multi_op(List<String> create_paths, List<String> delete_paths, List<Version> delete_versions, List<String> update_paths, List<String> update_versions, List<String> update_data); or struct op { int type; // OP_CREATE, OP_DELETE, OP_UPDATE String path; long version; // ignored for OP_CREATE, must be -1 to be ignored for OP_DELETE | OP_UPDATE char *data; // null for OP_DELETE } int zk_multi_op(List<op> operations); in the core API, wrapped with a builder in the Java client. Henry -- Henry Robinson Software Engineer Cloudera 415-994-6679
-
Re: need for more conditional write supportTed Dunning 2010-12-22, 18:29
This is quite similar to my other proposal except that I used polymorphism
instead of an enum. I would expect the wire format to look like what you specify. On Wed, Dec 22, 2010 at 1:00 AM, Henry Robinson <[EMAIL PROTECTED]> wrote: > A builder API is a bit of a departure from the core ZK API as it stands. > I'd > rather see something like: > > zk_multi_op(List<String> create_paths, List<String> delete_paths, > List<Version> delete_versions, List<String> update_paths, List<String> > update_versions, List<String> update_data); > |