Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - requiring zookeeper 3.4.x in 0.96


+
Jesse Yates 2012-04-18, 20:33
+
lars hofhansl 2012-04-18, 20:48
+
Todd Lipcon 2012-04-18, 21:49
+
Jesse Yates 2012-04-18, 22:29
Copy link to this message
-
Re: requiring zookeeper 3.4.x in 0.96
Todd Lipcon 2012-04-18, 22:34
On Wed, Apr 18, 2012 at 3:29 PM, Jesse Yates <[EMAIL PROTECTED]> wrote:
> Its makes dealing with a lot of the zk operations easier to reason about.
> Consider the 'create with parents' or the 'delete recursively' use cases -
> non-transactional guarantees means we have a lot of extra logic for
> checking under parents on creation, synchronous operations requiring extra
> checking/notification etc; in short, its a lot of logic for things that
> aren't, on a high-level, all that complicated (and are thought of
> transactionally).

But how much do we really do these things? Isn't it only when we
bootstrap a new cluster?

>
> These are the corner cases that are harder to find. For instance, what if
> the parent is there, but the child has been put there - I need to leave a
> watch and then do an update when children changed, and then read in all the
> children, which could be actually more than I'm looking for, which leads to
> a bunch of extra IO. If you can be sure that this goes away, we can get
> much simpler code - notify for the node and then read the children. If the
> children aren't there, they won't show up. this is just one possible
> example, but others can also be constructed.

You mean in the middle of a recursive op?

>
> It also can enable some really nice, new use cases. For instance, I've
> discussed doing a Paxos-like mechanism that is far more easily enabled by
> having transactional read-modify-write semantics for things like dynamic
> table configuration (not proposing this as a jira (yet), and don't want to
> get into if its a good idea, but rather just as an example for possible new
> uses).

ZK already has version-checking setData() calls which is enough for
optimistic concurrency... I don't get the "paxos-like mechanism" here
when ZK is already giving us consensus and atomic CAS.

-Todd

>
> On Wed, Apr 18, 2012 at 2:49 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> ZK 3.4.x still isn't marked "stable", so I'm a little hesitant to move
>> to requiring it just yet.
>>
>> What tangible benefit do we gain from the "transactional" methods?
>>
>> -Todd
>>
>> On Wed, Apr 18, 2012 at 1:48 PM, lars hofhansl <[EMAIL PROTECTED]>
>> wrote:
>> > +1
>> >
>> >
>> >
>> > ----- Original Message -----
>> > From: Jesse Yates <[EMAIL PROTECTED]>
>> > To: [EMAIL PROTECTED]
>> > Cc:
>> > Sent: Wednesday, April 18, 2012 1:33 PM
>> > Subject: requiring zookeeper 3.4.x in 0.96
>> >
>> > Hi all,
>> >
>> > I was working on some patches to make ZK a bit more sane using the
>> > transaction operations recently added (in jira:
>> > https://issues.apache.org/jira/browse/ZOOKEEPER-965
>> > <
>> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
>> >
>> > and in the api here:
>> >
>> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
>> ).
>> > This takes care of a lot of race conditions, etc. that are inherently
>> > present in zk.
>> >
>> > Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production
>> and
>> > instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
>> > client. Adding in transaction support will break this cross-version
>> > functionality without a way of falling back as ZK doesn't (afaik) support
>> > client-side look up of the server's version. Further, the fallback
>> > functionality would be very dangerous if we assume transactions but fall
>> > back to the non-transactional method; if we don't assume transactions,
>> then
>> > the transactional method is just wasted effort.
>> >
>> > I'm proposing that in the singularity (0.96), when we move to requiring
>> > Hadoop v1.0+ we also make it a requirement that people run ZooKeeper
>> v3.4+
>> >
>> > Thoughts?
>> >
>> > Thanks!
>> >
>> > -------------------
>> > Jesse Yates
>> > 240-888-2200
>> > @jesse_yates
>> > jyates.github.com
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>

--
Todd Lipcon
Software Engineer, Cloudera
+
Jesse Yates 2012-04-18, 22:45
+
Ted Yu 2012-04-18, 22:44
+
Stack 2012-04-18, 22:46
+
Andrew Purtell 2012-04-19, 01:41
+
Andrew Purtell 2012-04-19, 01:45