Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - requiring zookeeper 3.4.x in 0.96


Copy link to this message
-
Re: requiring zookeeper 3.4.x in 0.96
Jesse Yates 2012-04-18, 22:29
Its makes dealing with a lot of the zk operations easier to reason about.
Consider the 'create with parents' or the 'delete recursively' use cases -
non-transactional guarantees means we have a lot of extra logic for
checking under parents on creation, synchronous operations requiring extra
checking/notification etc; in short, its a lot of logic for things that
aren't, on a high-level, all that complicated (and are thought of
transactionally).

These are the corner cases that are harder to find. For instance, what if
the parent is there, but the child has been put there - I need to leave a
watch and then do an update when children changed, and then read in all the
children, which could be actually more than I'm looking for, which leads to
a bunch of extra IO. If you can be sure that this goes away, we can get
much simpler code - notify for the node and then read the children. If the
children aren't there, they won't show up. this is just one possible
example, but others can also be constructed.

It also can enable some really nice, new use cases. For instance, I've
discussed doing a Paxos-like mechanism that is far more easily enabled by
having transactional read-modify-write semantics for things like dynamic
table configuration (not proposing this as a jira (yet), and don't want to
get into if its a good idea, but rather just as an example for possible new
uses).

-------------------
Jesse Yates
240-888-2200
@jesse_yates
jyates.github.com
On Wed, Apr 18, 2012 at 2:49 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> ZK 3.4.x still isn't marked "stable", so I'm a little hesitant to move
> to requiring it just yet.
>
> What tangible benefit do we gain from the "transactional" methods?
>
> -Todd
>
> On Wed, Apr 18, 2012 at 1:48 PM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
> > +1
> >
> >
> >
> > ----- Original Message -----
> > From: Jesse Yates <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Cc:
> > Sent: Wednesday, April 18, 2012 1:33 PM
> > Subject: requiring zookeeper 3.4.x in 0.96
> >
> > Hi all,
> >
> > I was working on some patches to make ZK a bit more sane using the
> > transaction operations recently added (in jira:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-965
> > <
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> >
> > and in the api here:
> >
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> ).
> > This takes care of a lot of race conditions, etc. that are inherently
> > present in zk.
> >
> > Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production
> and
> > instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
> > client. Adding in transaction support will break this cross-version
> > functionality without a way of falling back as ZK doesn't (afaik) support
> > client-side look up of the server's version. Further, the fallback
> > functionality would be very dangerous if we assume transactions but fall
> > back to the non-transactional method; if we don't assume transactions,
> then
> > the transactional method is just wasted effort.
> >
> > I'm proposing that in the singularity (0.96), when we move to requiring
> > Hadoop v1.0+ we also make it a requirement that people run ZooKeeper
> v3.4+
> >
> > Thoughts?
> >
> > Thanks!
> >
> > -------------------
> > Jesse Yates
> > 240-888-2200
> > @jesse_yates
> > jyates.github.com
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>