Hi all,

I was working on some patches to make ZK a bit more sane using the
transaction operations recently added (in jira:
https://issues.apache.org/jira/browse/ZOOKEEPER-965
<http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29>
and in the api here:
http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29).
This takes care of a lot of race conditions, etc. that are inherently
present in zk.

Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production and
instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
client. Adding in transaction support will break this cross-version
functionality without a way of falling back as ZK doesn't (afaik) support
client-side look up of the server's version. Further, the fallback
functionality would be very dangerous if we assume transactions but fall
back to the non-transactional method; if we don't assume transactions, then
the transactional method is just wasted effort.

I'm proposing that in the singularity (0.96), when we move to requiring
Hadoop v1.0+ we also make it a requirement that people run ZooKeeper v3.4+

Thoughts?

Thanks!

-------------------
Jesse Yates
240-888-2200
@jesse_yates
jyates.github.com

+1

----- Original Message -----
From: Jesse Yates <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc:
Sent: Wednesday, April 18, 2012 1:33 PM
Subject: requiring zookeeper 3.4.x in 0.96

Hi all,

I was working on some patches to make ZK a bit more sane using the
transaction operations recently added (in jira:
https://issues.apache.org/jira/browse/ZOOKEEPER-965
<http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29>
and in the api here:
http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29).
This takes care of a lot of race conditions, etc. that are inherently
present in zk.

Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production and
instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
client. Adding in transaction support will break this cross-version
functionality without a way of falling back as ZK doesn't (afaik) support
client-side look up of the server's version. Further, the fallback
functionality would be very dangerous if we assume transactions but fall
back to the non-transactional method; if we don't assume transactions, then
the transactional method is just wasted effort.

I'm proposing that in the singularity (0.96), when we move to requiring
Hadoop v1.0+ we also make it a requirement that people run ZooKeeper v3.4+

Thoughts?

Thanks!

-------------------
Jesse Yates
240-888-2200
@jesse_yates
jyates.github.com

ZK 3.4.x still isn't marked "stable", so I'm a little hesitant to move
to requiring it just yet.

What tangible benefit do we gain from the "transactional" methods?

-Todd

On Wed, Apr 18, 2012 at 1:48 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> +1
>
>
>
> ----- Original Message -----
> From: Jesse Yates <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Wednesday, April 18, 2012 1:33 PM
> Subject: requiring zookeeper 3.4.x in 0.96
>
> Hi all,
>
> I was working on some patches to make ZK a bit more sane using the
> transaction operations recently added (in jira:
> https://issues.apache.org/jira/browse/ZOOKEEPER-965
> <http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29>
> and in the api here:
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29).
> This takes care of a lot of race conditions, etc. that are inherently
> present in zk.
>
> Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production and
> instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
> client. Adding in transaction support will break this cross-version
> functionality without a way of falling back as ZK doesn't (afaik) support
> client-side look up of the server's version. Further, the fallback
> functionality would be very dangerous if we assume transactions but fall
> back to the non-transactional method; if we don't assume transactions, then
> the transactional method is just wasted effort.
>
> I'm proposing that in the singularity (0.96), when we move to requiring
> Hadoop v1.0+ we also make it a requirement that people run ZooKeeper v3.4+
>
> Thoughts?
>
> Thanks!
>
> -------------------
> Jesse Yates
> 240-888-2200
> @jesse_yates
> jyates.github.com
>

--
Todd Lipcon
Software Engineer, Cloudera

Its makes dealing with a lot of the zk operations easier to reason about.
Consider the 'create with parents' or the 'delete recursively' use cases -
non-transactional guarantees means we have a lot of extra logic for
checking under parents on creation, synchronous operations requiring extra
checking/notification etc; in short, its a lot of logic for things that
aren't, on a high-level, all that complicated (and are thought of
transactionally).

These are the corner cases that are harder to find. For instance, what if
the parent is there, but the child has been put there - I need to leave a
watch and then do an update when children changed, and then read in all the
children, which could be actually more than I'm looking for, which leads to
a bunch of extra IO. If you can be sure that this goes away, we can get
much simpler code - notify for the node and then read the children. If the
children aren't there, they won't show up. this is just one possible
example, but others can also be constructed.

It also can enable some really nice, new use cases. For instance, I've
discussed doing a Paxos-like mechanism that is far more easily enabled by
having transactional read-modify-write semantics for things like dynamic
table configuration (not proposing this as a jira (yet), and don't want to
get into if its a good idea, but rather just as an example for possible new
uses).

-------------------
Jesse Yates
240-888-2200
@jesse_yates
jyates.github.com
On Wed, Apr 18, 2012 at 2:49 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> ZK 3.4.x still isn't marked "stable", so I'm a little hesitant to move
> to requiring it just yet.
>
> What tangible benefit do we gain from the "transactional" methods?
>
> -Todd
>
> On Wed, Apr 18, 2012 at 1:48 PM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
> > +1
> >
> >
> >
> > ----- Original Message -----
> > From: Jesse Yates <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Cc:
> > Sent: Wednesday, April 18, 2012 1:33 PM
> > Subject: requiring zookeeper 3.4.x in 0.96
> >
> > Hi all,
> >
> > I was working on some patches to make ZK a bit more sane using the
> > transaction operations recently added (in jira:
> > https://issues.apache.org/jira/browse/ZOOKEEPER-965
> > <
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> >
> > and in the api here:
> >
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> ).
> > This takes care of a lot of race conditions, etc. that are inherently
> > present in zk.
> >
> > Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production
> and
> > instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
> > client. Adding in transaction support will break this cross-version
> > functionality without a way of falling back as ZK doesn't (afaik) support
> > client-side look up of the server's version. Further, the fallback
> > functionality would be very dangerous if we assume transactions but fall
> > back to the non-transactional method; if we don't assume transactions,
> then
> > the transactional method is just wasted effort.
> >
> > I'm proposing that in the singularity (0.96), when we move to requiring
> > Hadoop v1.0+ we also make it a requirement that people run ZooKeeper
> v3.4+
> >
> > Thoughts?
> >
> > Thanks!
> >
> > -------------------
> > Jesse Yates
> > 240-888-2200
> > @jesse_yates
> > jyates.github.com
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

On Wed, Apr 18, 2012 at 3:29 PM, Jesse Yates <[EMAIL PROTECTED]> wrote:
> Its makes dealing with a lot of the zk operations easier to reason about.
> Consider the 'create with parents' or the 'delete recursively' use cases -
> non-transactional guarantees means we have a lot of extra logic for
> checking under parents on creation, synchronous operations requiring extra
> checking/notification etc; in short, its a lot of logic for things that
> aren't, on a high-level, all that complicated (and are thought of
> transactionally).

But how much do we really do these things? Isn't it only when we
bootstrap a new cluster?

>
> These are the corner cases that are harder to find. For instance, what if
> the parent is there, but the child has been put there - I need to leave a
> watch and then do an update when children changed, and then read in all the
> children, which could be actually more than I'm looking for, which leads to
> a bunch of extra IO. If you can be sure that this goes away, we can get
> much simpler code - notify for the node and then read the children. If the
> children aren't there, they won't show up. this is just one possible
> example, but others can also be constructed.

You mean in the middle of a recursive op?

>
> It also can enable some really nice, new use cases. For instance, I've
> discussed doing a Paxos-like mechanism that is far more easily enabled by
> having transactional read-modify-write semantics for things like dynamic
> table configuration (not proposing this as a jira (yet), and don't want to
> get into if its a good idea, but rather just as an example for possible new
> uses).

ZK already has version-checking setData() calls which is enough for
optimistic concurrency... I don't get the "paxos-like mechanism" here
when ZK is already giving us consensus and atomic CAS.

-Todd

>
> On Wed, Apr 18, 2012 at 2:49 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> ZK 3.4.x still isn't marked "stable", so I'm a little hesitant to move
>> to requiring it just yet.
>>
>> What tangible benefit do we gain from the "transactional" methods?
>>
>> -Todd
>>
>> On Wed, Apr 18, 2012 at 1:48 PM, lars hofhansl <[EMAIL PROTECTED]>
>> wrote:
>> > +1
>> >
>> >
>> >
>> > ----- Original Message -----
>> > From: Jesse Yates <[EMAIL PROTECTED]>
>> > To: [EMAIL PROTECTED]
>> > Cc:
>> > Sent: Wednesday, April 18, 2012 1:33 PM
>> > Subject: requiring zookeeper 3.4.x in 0.96
>> >
>> > Hi all,
>> >
>> > I was working on some patches to make ZK a bit more sane using the
>> > transaction operations recently added (in jira:
>> > https://issues.apache.org/jira/browse/ZOOKEEPER-965
>> > <
>> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
>> >
>> > and in the api here:
>> >
>> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
>> ).
>> > This takes care of a lot of race conditions, etc. that are inherently
>> > present in zk.
>> >
>> > Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production
>> and
>> > instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
>> > client. Adding in transaction support will break this cross-version
>> > functionality without a way of falling back as ZK doesn't (afaik) support
>> > client-side look up of the server's version. Further, the fallback
>> > functionality would be very dangerous if we assume transactions but fall
>> > back to the non-transactional method; if we don't assume transactions,
>> then
>> > the transactional method is just wasted effort.
>> >
>> > I'm proposing that in the singularity (0.96), when we move to requiring
>> > Hadoop v1.0+ we also make it a requirement that people run ZooKeeper
>> v3.4+
>> >
>> > Thoughts?
>> >
>> > Thanks!
>> >
>> > -------------------
>> > Jesse Yates
>> > 240-888-2200
>> > @jesse_yates
>> > jyates.github.com
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>

--
Todd Lipcon
Software Engineer, Cloudera

I want to mention the work Devaraj D is doing in HBASE-5732: merging the
secure RPC engine.

I think the implication of that work is zookeeper 3.4.x would be required
for 0.96

Cheers

On Wed, Apr 18, 2012 at 1:33 PM, Jesse Yates <[EMAIL PROTECTED]>wrote:

> Hi all,
>
> I was working on some patches to make ZK a bit more sane using the
> transaction operations recently added (in jira:
> https://issues.apache.org/jira/browse/ZOOKEEPER-965
> <
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> >
> and in the api here:
>
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> ).
> This takes care of a lot of race conditions, etc. that are inherently
> present in zk.
>
> Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production and
> instead allow users to run down to at least 3.3.x with the hbase's  3.4.x
> client. Adding in transaction support will break this cross-version
> functionality without a way of falling back as ZK doesn't (afaik) support
> client-side look up of the server's version. Further, the fallback
> functionality would be very dangerous if we assume transactions but fall
> back to the non-transactional method; if we don't assume transactions, then
> the transactional method is just wasted effort.
>
> I'm proposing that in the singularity (0.96), when we move to requiring
> Hadoop v1.0+ we also make it a requirement that people run ZooKeeper v3.4+
>
> Thoughts?
>
> Thanks!
>
> -------------------
> Jesse Yates
> 240-888-2200
> @jesse_yates
> jyates.github.com
>

-------------------
Jesse Yates
240-888-2200
@jesse_yates
jyates.github.com
On Wed, Apr 18, 2012 at 3:34 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Wed, Apr 18, 2012 at 3:29 PM, Jesse Yates <[EMAIL PROTECTED]>
> wrote:
> > Its makes dealing with a lot of the zk operations easier to reason about.
> > Consider the 'create with parents' or the 'delete recursively' use cases
> -
> > non-transactional guarantees means we have a lot of extra logic for
> > checking under parents on creation, synchronous operations requiring
> extra
> > checking/notification etc; in short, its a lot of logic for things that
> > aren't, on a high-level, all that complicated (and are thought of
> > transactionally).
>
> But how much do we really do these things? Isn't it only when we
> bootstrap a new cluster?
>

Depends on the operation - the replication code has a lot of reliance of
moving zk nodes around and the hfile backup stuff that is coming also does
dynamic znode allocation.
>
>
>
 >
> > These are the corner cases that are harder to find. For instance, what if
> > the parent is there, but the child has been put there - I need to leave a
> > watch and then do an update when children changed, and then read in all
> the
> > children, which could be actually more than I'm looking for, which leads
> to
> > a bunch of extra IO. If you can be sure that this goes away, we can get
> > much simpler code - notify for the node and then read the children. If
> the
> > children aren't there, they won't show up. this is just one possible
> > example, but others can also be constructed.
>
> You mean in the middle of a recursive op?
>

Yeah, exactly. Its an annoying use case and one easily messed up.
>
> >
> > It also can enable some really nice, new use cases. For instance, I've
> > discussed doing a Paxos-like mechanism that is far more easily enabled by
> > having transactional read-modify-write semantics for things like dynamic
> > table configuration (not proposing this as a jira (yet), and don't want
> to
> > get into if its a good idea, but rather just as an example for possible
> new
> > uses).
>
> ZK already has version-checking setData() calls which is enough for
> optimistic concurrency... I don't get the "paxos-like mechanism" here
> when ZK is already giving us consensus and atomic CAS.
>

Oh yeah, you are right. A little confused there on where the ZK version
stuff comes from (since you don't actually set the version, but are just
given it by zk)
>
> -Todd
>
> >
> > On Wed, Apr 18, 2012 at 2:49 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >
> >> ZK 3.4.x still isn't marked "stable", so I'm a little hesitant to move
> >> to requiring it just yet.
> >>
> >> What tangible benefit do we gain from the "transactional" methods?
> >>
> >> -Todd
> >>
> >> On Wed, Apr 18, 2012 at 1:48 PM, lars hofhansl <[EMAIL PROTECTED]>
> >> wrote:
> >> > +1
> >> >
> >> >
> >> >
> >> > ----- Original Message -----
> >> > From: Jesse Yates <[EMAIL PROTECTED]>
> >> > To: [EMAIL PROTECTED]
> >> > Cc:
> >> > Sent: Wednesday, April 18, 2012 1:33 PM
> >> > Subject: requiring zookeeper 3.4.x in 0.96
> >> >
> >> > Hi all,
> >> >
> >> > I was working on some patches to make ZK a bit more sane using the
> >> > transaction operations recently added (in jira:
> >> > https://issues.apache.org/jira/browse/ZOOKEEPER-965
> >> > <
> >>
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> >> >
> >> > and in the api here:
> >> >
> >>
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#transaction%28%29
> >> ).
> >> > This takes care of a lot of race conditions, etc. that are inherently
> >> > present in zk.
> >> >
> >> > Currently we (afaik) are not requiring ZooKeeper v.3.4.x in production
> >> and
> >> > instead allow users to run down to at least 3.3.x with the hbase's
>  3.4.x
> >> > client. Adding in transaction support will break this cross-version
> >> > functionality without a way of falling back as ZK doesn't (afaik)

On Wed, Apr 18, 2012 at 3:44 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> I want to mention the work Devaraj D is doing in HBASE-5732: merging the
> secure RPC engine.
>
> I think the implication of that work is zookeeper 3.4.x would be required
> for 0.96
>

Only at runtime if you run secure hbase.
St.Ack

> Only at runtime if you run secure hbase.
> St.Ack
Correct.

Additionally there are two levels of HBase security, if you will:

1) RPC security: secure RPC engine, Kerberos authentication via SASL, optional wire encryption via SASL
2) Coprocessor based access control at the HBase column family level

You can opt for none, or #1 and not need to run ZK 3.4.x.

Only if you opt for #1 and #2 (#2 requires #1), then you are strongly recommended to run ZK 3.4.x.
Best regards,
    - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

----- Original Message -----
> From: Stack <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Thursday, April 19, 2012 6:46 AM
> Subject: Re: requiring zookeeper 3.4.x in 0.96
>
> On Wed, Apr 18, 2012 at 3:44 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>  I want to mention the work Devaraj D is doing in HBASE-5732: merging the
>>  secure RPC engine.
>>
>>  I think the implication of that work is zookeeper 3.4.x would be required
>>  for 0.96
>>
>
> Only at runtime if you run secure hbase.
> St.Ack
>

So, in case it's not clear, Deveraj's work doesn't involve ZK. Secure RPC doesn't care about ZK version.You might care about securing ZK with SASL auth if you are also running secure Hadoop (and HBase) -- I would -- but that would be up to you.
Best regards,
    - Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

----- Original Message -----
> From: Andrew Purtell <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Cc:
> Sent: Thursday, April 19, 2012 9:41 AM
> Subject: Re: requiring zookeeper 3.4.x in 0.96
>
>>  Only at runtime if you run secure hbase.
>>  St.Ack
>
>
> Correct.
>
> Additionally there are two levels of HBase security, if you will:
>
> 1) RPC security: secure RPC engine, Kerberos authentication via SASL, optional
> wire encryption via SASL
>
>
> 2) Coprocessor based access control at the HBase column family level
>
> You can opt for none, or #1 and not need to run ZK 3.4.x.
>
> Only if you opt for #1 and #2 (#2 requires #1), then you are strongly
> recommended to run ZK 3.4.x.
> Best regards,
>
>
>     - Andy
>
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via
> Tom White)
>
>
>
> ----- Original Message -----
>>  From: Stack <[EMAIL PROTECTED]>
>>  To: [EMAIL PROTECTED]
>>  Cc:
>>  Sent: Thursday, April 19, 2012 6:46 AM
>>  Subject: Re: requiring zookeeper 3.4.x in 0.96
>>
>>  On Wed, Apr 18, 2012 at 3:44 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>   I want to mention the work Devaraj D is doing in HBASE-5732: merging
> the
>>>   secure RPC engine.
>>>
>>>   I think the implication of that work is zookeeper 3.4.x would be
> required
>>>   for 0.96
>>>
>>
>>  Only at runtime if you run secure hbase.
>>  St.Ack
>>
>