Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> requiring zookeeper 3.4.x in 0.96


+
Jesse Yates 2012-04-18, 20:33
+
lars hofhansl 2012-04-18, 20:48
+
Todd Lipcon 2012-04-18, 21:49
+
Jesse Yates 2012-04-18, 22:29
Copy link to this message
-
Re: requiring zookeeper 3.4.x in 0.96
On Wed, Apr 18, 2012 at 3:29 PM, Jesse Yates <[EMAIL PROTECTED]> wrote:
> Its makes dealing with a lot of the zk operations easier to reason about.
> Consider the 'create with parents' or the 'delete recursively' use cases -
> non-transactional guarantees means we have a lot of extra logic for
> checking under parents on creation, synchronous operations requiring extra
> checking/notification etc; in short, its a lot of logic for things that
> aren't, on a high-level, all that complicated (and are thought of
> transactionally).

But how much do we really do these things? Isn't it only when we
bootstrap a new cluster?

>
> These are the corner cases that are harder to find. For instance, what if
> the parent is there, but the child has been put there - I need to leave a
> watch and then do an update when children changed, and then read in all the
> children, which could be actually more than I'm looking for, which leads to
> a bunch of extra IO. If you can be sure that this goes away, we can get
> much simpler code - notify for the node and then read the children. If the
> children aren't there, they won't show up. this is just one possible
> example, but others can also be constructed.

You mean in the middle of a recursive op?

>
> It also can enable some really nice, new use cases. For instance, I've
> discussed doing a Paxos-like mechanism that is far more easily enabled by
> having transactional read-modify-write semantics for things like dynamic
> table configuration (not proposing this as a jira (yet), and don't want to
> get into if its a good idea, but rather just as an example for possible new
> uses).

ZK already has version-checking setData() calls which is enough for
optimistic concurrency... I don't get the "paxos-like mechanism" here
when ZK is already giving us consensus and atomic CAS.

-Todd

>
> On Wed, Apr 18, 2012 at 2:49 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> ZK 3.4.x still isn't marked "stable", so I'm a little hesitant to move
>> to requiring it just yet.
>>
>> What tangible benefit do we gain from the "transactional" methods?
>>
>> -Todd
>>
>> On Wed, Apr 18, 2012 at 1:48 PM, lars hofhansl <[EMAIL PROTECTED]>
>> wrote:
>> > +1
>> >
>> >
>> >
>> > ----- Original Message -----
>> > From: Jesse Yates <[EMAIL PROTECTED]>
>> > To: [EMAIL PROTECTED]
>> > Cc:
>> > Sent: Wednesday, April 18, 2012 1:33 PM
>> > Subject: requiring zookeeper 3.4.x in 0.96
>> >
>> > Hi all,
>> >
>> > I was working on some patches to make ZK a bit more sane using the
>> > transaction operations recently added (in jira:
>> > https://issues.apache.org/jira/browse/ZOOKEEPER-965
>> > <
>> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
>> >
>> > and in the api here:
>> >
>> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
>> ).
>> > This takes care of a lot of race conditions, etc. that are inherently
>> > present in zk.
>> >
>> > Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production
>> and
>> > instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
>> > client. Adding in transaction support will break this cross-version
>> > functionality without a way of falling back as ZK doesn't (afaik) support
>> > client-side look up of the server's version. Further, the fallback
>> > functionality would be very dangerous if we assume transactions but fall
>> > back to the non-transactional method; if we don't assume transactions,
>> then
>> > the transactional method is just wasted effort.
>> >
>> > I'm proposing that in the singularity (0.96), when we move to requiring
>> > Hadoop v1.0+ we also make it a requirement that people run ZooKeeper
>> v3.4+
>> >
>> > Thoughts?
>> >
>> > Thanks!
>> >
>> > -------------------
>> > Jesse Yates
>> > 240-888-2200
>> > @jesse_yates
>> > jyates.github.com
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>

--
Todd Lipcon
Software Engineer, Cloudera
+
Jesse Yates 2012-04-18, 22:45
+
Ted Yu 2012-04-18, 22:44
+
Stack 2012-04-18, 22:46
+
Andrew Purtell 2012-04-19, 01:41
+
Andrew Purtell 2012-04-19, 01:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB