HBase 0.92 is using zookeeper 3.4.2

Maybe some of you have seen this JIRA
https://issues.apache.org/jira/browse/ZOOKEEPER-1367
It looks like a serious issue.

Cheers

At one point I had proposed making the ZK dependency switch only for
the security profile in the pom. The ZK 3.4.x series has been buggy so
far  - I'm sure it will stabilize within month or two, but I'd be +1
on reverting the non-secure build to 3.3.x in the meantime.

-Todd

On Thu, Jan 26, 2012 at 8:25 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> HBase 0.92 is using zookeeper 3.4.2
>
> Maybe some of you have seen this JIRA
> https://issues.apache.org/jira/browse/ZOOKEEPER-1367
> It looks like a serious issue.
>
> Cheers

--
Todd Lipcon
Software Engineer, Cloudera

That's what we have done for internal repository.

Some of the bugs in 3.4.x are hard to reproduce, track down and fix.

Of course, Gary and Andrew's opinions are important.

On Fri, Jan 27, 2012 at 7:23 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> At one point I had proposed making the ZK dependency switch only for
> the security profile in the pom. The ZK 3.4.x series has been buggy so
> far  - I'm sure it will stabilize within month or two, but I'd be +1
> on reverting the non-secure build to 3.3.x in the meantime.
>
> -Todd
>
> On Thu, Jan 26, 2012 at 8:25 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > HBase 0.92 is using zookeeper 3.4.2
> >
> > Maybe some of you have seen this JIRA
> > https://issues.apache.org/jira/browse/ZOOKEEPER-1367
> > It looks like a serious issue.
> >
> > Cheers
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

As I recall, there were other API changes in zk 3.3 -> 3.4 that would
make reverting a bit more complicated.  Like the change of
NIOServerCnxn.Factory -> NIOServerCnxnFactory (refactor to top level
class).  So reverting while keeping 3.4 usage for security would
require more work to put in place some kind of shim layer.

In any case, security is meaningless without ZK 3.4, so I am not in
favor of reverting.  I haven't been tracking 3.4 development closely,
so I don't know how much pain bugs in that release have been causing.
But 3.3 has had issues too.  I was just bit by ZOOKEEPER-1208 last
week on a running cluster.  Of course this issue is fixed in 3.3.4 and
3.4.0.  But that would by my opinion for any current issues we're
seeing with 3.4 as well -- let's try to get them fixed and move on
instead of putting effort into backtracking for a temporary solution.

--gh
On Fri, Jan 27, 2012 at 7:50 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> That's what we have done for internal repository.
>
> Some of the bugs in 3.4.x are hard to reproduce, track down and fix.
>
> Of course, Gary and Andrew's opinions are important.
>
> On Fri, Jan 27, 2012 at 7:23 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> At one point I had proposed making the ZK dependency switch only for
>> the security profile in the pom. The ZK 3.4.x series has been buggy so
>> far  - I'm sure it will stabilize within month or two, but I'd be +1
>> on reverting the non-secure build to 3.3.x in the meantime.
>>
>> -Todd
>>
>> On Thu, Jan 26, 2012 at 8:25 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > HBase 0.92 is using zookeeper 3.4.2
>> >
>> > Maybe some of you have seen this JIRA
>> > https://issues.apache.org/jira/browse/ZOOKEEPER-1367
>> > It looks like a serious issue.
>> >
>> > Cheers
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>

If you dig in on ZOOKEEPER-1367, this is a custom ZooKeeper embedding in a product, not a deployment scenario that one would see with HBase. I don't want to overestimate or underestimate the importance of the issue. Currently it is under investigation and the ZK folks haven't gotten to the bottom of it. Making a decision based on this one JIRA seems premature.

> In any case, security is meaningless without ZK 3.4, so I am not in
> favor of reverting. 
Likewise. 

Whomever reverts the main build to ZK to 3.3 while retaining 3.4 for security would have to add shims for the NIO server constructor. There is also a problematic Enum change. 
 
Best regards,

     - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Gary Helmling <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Friday, January 27, 2012 9:16 AM
> Subject: Re: rethinking zookeeper version
>
> As I recall, there were other API changes in zk 3.3 -> 3.4 that would
> make reverting a bit more complicated.  Like the change of
> NIOServerCnxn.Factory -> NIOServerCnxnFactory (refactor to top level
> class).  So reverting while keeping 3.4 usage for security would
> require more work to put in place some kind of shim layer.
>
> In any case, security is meaningless without ZK 3.4, so I am not in
> favor of reverting.  I haven't been tracking 3.4 development closely,
> so I don't know how much pain bugs in that release have been causing.
> But 3.3 has had issues too.  I was just bit by ZOOKEEPER-1208 last
> week on a running cluster.  Of course this issue is fixed in 3.3.4 and
> 3.4.0.  But that would by my opinion for any current issues we're
> seeing with 3.4 as well -- let's try to get them fixed and move on
> instead of putting effort into backtracking for a temporary solution.
>
> --gh
>
>
> On Fri, Jan 27, 2012 at 7:50 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>  That's what we have done for internal repository.
>>
>>  Some of the bugs in 3.4.x are hard to reproduce, track down and fix.
>>
>>  Of course, Gary and Andrew's opinions are important.
>>
>>  On Fri, Jan 27, 2012 at 7:23 AM, Todd Lipcon <[EMAIL PROTECTED]>
> wrote:
>>
>>>  At one point I had proposed making the ZK dependency switch only for
>>>  the security profile in the pom. The ZK 3.4.x series has been buggy so
>>>  far  - I'm sure it will stabilize within month or two, but I'd
> be +1
>>>  on reverting the non-secure build to 3.3.x in the meantime.
>>>
>>>  -Todd
>>>
>>>  On Thu, Jan 26, 2012 at 8:25 PM, Ted Yu <[EMAIL PROTECTED]>
> wrote:
>>>  > HBase 0.92 is using zookeeper 3.4.2
>>>  >
>>>  > Maybe some of you have seen this JIRA
>>>  > https://issues.apache.org/jira/browse/ZOOKEEPER-1367
>>>  > It looks like a serious issue.
>>>  >
>>>  > Cheers
>>>
>>>
>>>
>>>  --
>>>  Todd Lipcon
>>>  Software Engineer, Cloudera
>>>
>

I agree with what Andy and Gary said.

In the future when we encounter incompatible API changes in a very
important Apache project which HBase depends heavily, we do need to
consider providing shim so that we have some room to accommodate different
releases of the underlying project in the (short) period after new release
of HBase.

Cheers

On Fri, Jan 27, 2012 at 10:14 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> If you dig in on ZOOKEEPER-1367, this is a custom ZooKeeper embedding in a
> product, not a deployment scenario that one would see with HBase. I don't
> want to overestimate or underestimate the importance of the issue.
> Currently it is under investigation and the ZK folks haven't gotten to the
> bottom of it. Making a decision based on this one JIRA seems premature.
>
> > In any case, security is meaningless without ZK 3.4, so I am not in
> > favor of reverting.
>
>
> Likewise.
>
> Whomever reverts the main build to ZK to 3.3 while retaining 3.4 for
> security would have to add shims for the NIO server constructor. There is
> also a problematic Enum change.
>
> Best regards,
>
>      - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> ----- Original Message -----
> > From: Gary Helmling <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Cc:
> > Sent: Friday, January 27, 2012 9:16 AM
> > Subject: Re: rethinking zookeeper version
> >
> > As I recall, there were other API changes in zk 3.3 -> 3.4 that would
> > make reverting a bit more complicated.  Like the change of
> > NIOServerCnxn.Factory -> NIOServerCnxnFactory (refactor to top level
> > class).  So reverting while keeping 3.4 usage for security would
> > require more work to put in place some kind of shim layer.
> >
> > In any case, security is meaningless without ZK 3.4, so I am not in
> > favor of reverting.  I haven't been tracking 3.4 development closely,
> > so I don't know how much pain bugs in that release have been causing.
> > But 3.3 has had issues too.  I was just bit by ZOOKEEPER-1208 last
> > week on a running cluster.  Of course this issue is fixed in 3.3.4 and
> > 3.4.0.  But that would by my opinion for any current issues we're
> > seeing with 3.4 as well -- let's try to get them fixed and move on
> > instead of putting effort into backtracking for a temporary solution.
> >
> > --gh
> >
> >
> > On Fri, Jan 27, 2012 at 7:50 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>  That's what we have done for internal repository.
> >>
> >>  Some of the bugs in 3.4.x are hard to reproduce, track down and fix.
> >>
> >>  Of course, Gary and Andrew's opinions are important.
> >>
> >>  On Fri, Jan 27, 2012 at 7:23 AM, Todd Lipcon <[EMAIL PROTECTED]>
> > wrote:
> >>
> >>>  At one point I had proposed making the ZK dependency switch only for
> >>>  the security profile in the pom. The ZK 3.4.x series has been buggy so
> >>>  far  - I'm sure it will stabilize within month or two, but I'd
> > be +1
> >>>  on reverting the non-secure build to 3.3.x in the meantime.
> >>>
> >>>  -Todd
> >>>
> >>>  On Thu, Jan 26, 2012 at 8:25 PM, Ted Yu <[EMAIL PROTECTED]>
> > wrote:
> >>>  > HBase 0.92 is using zookeeper 3.4.2
> >>>  >
> >>>  > Maybe some of you have seen this JIRA
> >>>  > https://issues.apache.org/jira/browse/ZOOKEEPER-1367
> >>>  > It looks like a serious issue.
> >>>  >
> >>>  > Cheers
> >>>
> >>>
> >>>
> >>>  --
> >>>  Todd Lipcon
> >>>  Software Engineer, Cloudera
> >>>
> >
>

Ted,

If I recall correctly, we did consider it. You were welcome at the time to make this comment on the appropriate JIRA. Maybe that would have changed the decision.

Shims are ugly hacks as a rule.

We also may have overreached trying to be compatible with CDH3, ASF 1.0, and 0.22, and 0.23. I know the MR tests are broken against 0.23. Also a change for 1.0 broke against 0.23 if I recall correctly. If I want to get the tests working against 0.23 there's some work to do. Which may break against 1.0, requiring more work, which may break against ... 

And what about subtle changes that don't break compilation but require a combinatorial explosion of test configurations to identify at runtime?

Where does that end?

My prediction: A decision to support one version of ZooKeeper, and core, with everything else YMMV.

Best regards,

     - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Ted Yu <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]; Andrew Purtell <[EMAIL PROTECTED]>
> Cc:
> Sent: Friday, January 27, 2012 10:26 AM
> Subject: Re: rethinking zookeeper version
>
> I agree with what Andy and Gary said.
>
> In the future when we encounter incompatible API changes in a very
> important Apache project which HBase depends heavily, we do need to
> consider providing shim so that we have some room to accommodate different
> releases of the underlying project in the (short) period after new release
> of HBase.
>
> Cheers
>
> On Fri, Jan 27, 2012 at 10:14 AM, Andrew Purtell
> <[EMAIL PROTECTED]>wrote:
>
>>  If you dig in on ZOOKEEPER-1367, this is a custom ZooKeeper embedding in a
>>  product, not a deployment scenario that one would see with HBase. I
> don't
>>  want to overestimate or underestimate the importance of the issue.
>>  Currently it is under investigation and the ZK folks haven't gotten to
> the
>>  bottom of it. Making a decision based on this one JIRA seems premature.
>>
>>  > In any case, security is meaningless without ZK 3.4, so I am not in
>>  > favor of reverting.
>>
>>
>>  Likewise.
>>
>>  Whomever reverts the main build to ZK to 3.3 while retaining 3.4 for
>>  security would have to add shims for the NIO server constructor. There is
>>  also a problematic Enum change.
>>
>>  Best regards,
>>
>>       - Andy
>>
>>  Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>  (via Tom White)
>>
>>
>>  ----- Original Message -----
>>  > From: Gary Helmling <[EMAIL PROTECTED]>
>>  > To: [EMAIL PROTECTED]
>>  > Cc:
>>  > Sent: Friday, January 27, 2012 9:16 AM
>>  > Subject: Re: rethinking zookeeper version
>>  >
>>  > As I recall, there were other API changes in zk 3.3 -> 3.4 that
> would
>>  > make reverting a bit more complicated.  Like the change of
>>  > NIOServerCnxn.Factory -> NIOServerCnxnFactory (refactor to top
> level
>>  > class).  So reverting while keeping 3.4 usage for security would
>>  > require more work to put in place some kind of shim layer.
>>  >
>>  > In any case, security is meaningless without ZK 3.4, so I am not in
>>  > favor of reverting.  I haven't been tracking 3.4 development
> closely,
>>  > so I don't know how much pain bugs in that release have been
> causing.
>>  > But 3.3 has had issues too.  I was just bit by ZOOKEEPER-1208 last
>>  > week on a running cluster.  Of course this issue is fixed in 3.3.4 and
>>  > 3.4.0.  But that would by my opinion for any current issues we're
>>  > seeing with 3.4 as well -- let's try to get them fixed and move on
>>  > instead of putting effort into backtracking for a temporary solution.
>>  >
>>  > --gh
>>  >
>>  >
>>  > On Fri, Jan 27, 2012 at 7:50 AM, Ted Yu <[EMAIL PROTECTED]>
> wrote:
>>  >>  That's what we have done for internal repository.
>>  >>
>>  >>  Some of the bugs in 3.4.x are hard to reproduce, track down and
> fix.
>>  >>
>>  >>  Of course, Gary and Andrew's opinions are important.
>>  >>
>>  >>  On Fri, Jan 27, 2012 at 7:23 AM, Todd Lipcon

On Fri, Jan 27, 2012 at 11:50 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> Shims are ugly hacks as a rule.
>

3.4.x zk client works against a 3.3.x ensemble (its supposed to and
seemed to in my basic tests) so why do we even need a shim  in this
particular case.

The subject for this email is bit chicken little.

St.Ack

Looks like Camille has found something for ZOOKEEPER-1367:
bq. Anyway, I think the problem is not recording the create session to the
txn log.

FYI

On Fri, Jan 27, 2012 at 10:14 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> If you dig in on ZOOKEEPER-1367, this is a custom ZooKeeper embedding in a
> product, not a deployment scenario that one would see with HBase. I don't
> want to overestimate or underestimate the importance of the issue.
> Currently it is under investigation and the ZK folks haven't gotten to the
> bottom of it. Making a decision based on this one JIRA seems premature.
>
> > In any case, security is meaningless without ZK 3.4, so I am not in
> > favor of reverting.
>
>
> Likewise.
>
> Whomever reverts the main build to ZK to 3.3 while retaining 3.4 for
> security would have to add shims for the NIO server constructor. There is
> also a problematic Enum change.
>
> Best regards,
>
>      - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> ----- Original Message -----
> > From: Gary Helmling <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Cc:
> > Sent: Friday, January 27, 2012 9:16 AM
> > Subject: Re: rethinking zookeeper version
> >
> > As I recall, there were other API changes in zk 3.3 -> 3.4 that would
> > make reverting a bit more complicated.  Like the change of
> > NIOServerCnxn.Factory -> NIOServerCnxnFactory (refactor to top level
> > class).  So reverting while keeping 3.4 usage for security would
> > require more work to put in place some kind of shim layer.
> >
> > In any case, security is meaningless without ZK 3.4, so I am not in
> > favor of reverting.  I haven't been tracking 3.4 development closely,
> > so I don't know how much pain bugs in that release have been causing.
> > But 3.3 has had issues too.  I was just bit by ZOOKEEPER-1208 last
> > week on a running cluster.  Of course this issue is fixed in 3.3.4 and
> > 3.4.0.  But that would by my opinion for any current issues we're
> > seeing with 3.4 as well -- let's try to get them fixed and move on
> > instead of putting effort into backtracking for a temporary solution.
> >
> > --gh
> >
> >
> > On Fri, Jan 27, 2012 at 7:50 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>  That's what we have done for internal repository.
> >>
> >>  Some of the bugs in 3.4.x are hard to reproduce, track down and fix.
> >>
> >>  Of course, Gary and Andrew's opinions are important.
> >>
> >>  On Fri, Jan 27, 2012 at 7:23 AM, Todd Lipcon <[EMAIL PROTECTED]>
> > wrote:
> >>
> >>>  At one point I had proposed making the ZK dependency switch only for
> >>>  the security profile in the pom. The ZK 3.4.x series has been buggy so
> >>>  far  - I'm sure it will stabilize within month or two, but I'd
> > be +1
> >>>  on reverting the non-secure build to 3.3.x in the meantime.
> >>>
> >>>  -Todd
> >>>
> >>>  On Thu, Jan 26, 2012 at 8:25 PM, Ted Yu <[EMAIL PROTECTED]>
> > wrote:
> >>>  > HBase 0.92 is using zookeeper 3.4.2
> >>>  >
> >>>  > Maybe some of you have seen this JIRA
> >>>  > https://issues.apache.org/jira/browse/ZOOKEEPER-1367
> >>>  > It looks like a serious issue.
> >>>  >
> >>>  > Cheers
> >>>
> >>>
> >>>
> >>>  --
> >>>  Todd Lipcon
> >>>  Software Engineer, Cloudera
> >>>
> >
>

I believe this fixed some HBase MR test issues when run against 0.23
https://issues.apache.org/jira/browse/HBASE-5212 (but is incompatible with
1.0.0)

FYI, There were some semantics changes in Hadoop 0.23.0 that broke some of
the tests with HBase 0.92.0 on it.  Here's jiras for those interested.

Some other HBase 0.92 test failures related fixed in Hadoop 0.23.x.
https://issues.apache.org/jira/browse/HDFS-2838
https://issues.apache.org/jira/browse/HADOOP-7997

Jon

On Fri, Jan 27, 2012 at 11:50 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> Ted,
>
> If I recall correctly, we did consider it. You were welcome at the time to
> make this comment on the appropriate JIRA. Maybe that would have changed
> the decision.
>
> Shims are ugly hacks as a rule.
>
> We also may have overreached trying to be compatible with CDH3, ASF 1.0,
> and 0.22, and 0.23. I know the MR tests are broken against 0.23. Also a
> change for 1.0 broke against 0.23 if I recall correctly. If I want to get
> the tests working against 0.23 there's some work to do. Which may break
> against 1.0, requiring more work, which may break against ...
>
> And what about subtle changes that don't break compilation but require a
> combinatorial explosion of test configurations to identify at runtime?
>
> Where does that end?
>
> My prediction: A decision to support one version of ZooKeeper, and core,
> with everything else YMMV.
>
> Best regards,
>
>      - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
>
> ----- Original Message -----
> > From: Ted Yu <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]; Andrew Purtell <[EMAIL PROTECTED]>
> > Cc:
> > Sent: Friday, January 27, 2012 10:26 AM
> > Subject: Re: rethinking zookeeper version
> >
> > I agree with what Andy and Gary said.
> >
> > In the future when we encounter incompatible API changes in a very
> > important Apache project which HBase depends heavily, we do need to
> > consider providing shim so that we have some room to accommodate
> different
> > releases of the underlying project in the (short) period after new
> release
> > of HBase.
> >
> > Cheers
> >
> > On Fri, Jan 27, 2012 at 10:14 AM, Andrew Purtell
> > <[EMAIL PROTECTED]>wrote:
> >
> >>  If you dig in on ZOOKEEPER-1367, this is a custom ZooKeeper embedding
> in a
> >>  product, not a deployment scenario that one would see with HBase. I
> > don't
> >>  want to overestimate or underestimate the importance of the issue.
> >>  Currently it is under investigation and the ZK folks haven't gotten to
> > the
> >>  bottom of it. Making a decision based on this one JIRA seems premature.
> >>
> >>  > In any case, security is meaningless without ZK 3.4, so I am not in
> >>  > favor of reverting.
> >>
> >>
> >>  Likewise.
> >>
> >>  Whomever reverts the main build to ZK to 3.3 while retaining 3.4 for
> >>  security would have to add shims for the NIO server constructor. There
> is
> >>  also a problematic Enum change.
> >>
> >>  Best regards,
> >>
> >>       - Andy
> >>
> >>  Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> >>  (via Tom White)
> >>
> >>
> >>  ----- Original Message -----
> >>  > From: Gary Helmling <[EMAIL PROTECTED]>
> >>  > To: [EMAIL PROTECTED]
> >>  > Cc:
> >>  > Sent: Friday, January 27, 2012 9:16 AM
> >>  > Subject: Re: rethinking zookeeper version
> >>  >
> >>  > As I recall, there were other API changes in zk 3.3 -> 3.4 that
> > would
> >>  > make reverting a bit more complicated.  Like the change of
> >>  > NIOServerCnxn.Factory -> NIOServerCnxnFactory (refactor to top
> > level
> >>  > class).  So reverting while keeping 3.4 usage for security would
> >>  > require more work to put in place some kind of shim layer.
> >>  >
> >>  > In any case, security is meaningless without ZK 3.4, so I am not in
> >>  > favor of reverting.  I haven't been tracking 3.4 development
> > closely,
> >>  > so I don't know how much pain bugs in that release have been
> > causing.

// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// [EMAIL PROTECTED]