Hello,

As HBase installation base is getting bigger, we are ready to work on the
wire compatibility issue.
The goal is to make HBase easier for operators to upgrade, while it is also
easier for developers to
enhance, re-architect if necessary.

The attached is a proposal we came up.  We'd like to start with two phases:

Phase 1: Compatibility between client applications and HBase clusters
Phase 2: HBase cluster rolling upgrade within same major version

Could you please review?

Thanks,
Jimmy

Can you post on wiki ?

Attachment stripped.

On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:

> Hello,
>
> As HBase installation base is getting bigger, we are ready to work on the
> wire compatibility issue.
> The goal is to make HBase easier for operators to upgrade, while it is
> also easier for developers to
> enhance, re-architect if necessary.
>
> The attached is a proposal we came up.  We'd like to start with two phases:
>
> Phase 1: Compatibility between client applications and HBase clusters
> Phase 2: HBase cluster rolling upgrade within same major version
>
> Could you please review?
>
> Thanks,
> Jimmy
>

I posted the proposal on wiki:

http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility

Thanks,
Jimmy

On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Can you post on wiki ?
>
> Attachment stripped.
>
> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > As HBase installation base is getting bigger, we are ready to work on the
> > wire compatibility issue.
> > The goal is to make HBase easier for operators to upgrade, while it is
> > also easier for developers to
> > enhance, re-architect if necessary.
> >
> > The attached is a proposal we came up.  We'd like to start with two
> phases:
> >
> > Phase 1: Compatibility between client applications and HBase clusters
> > Phase 2: HBase cluster rolling upgrade within same major version
> >
> > Could you please review?
> >
> > Thanks,
> > Jimmy
> >
>

What's the role of HBASE-5306 in your proposal ?

Thanks

On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:

> I posted the proposal on wiki:
>
> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>
> Thanks,
> Jimmy
>
> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > Can you post on wiki ?
> >
> > Attachment stripped.
> >
> > On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hello,
> > >
> > > As HBase installation base is getting bigger, we are ready to work on
> the
> > > wire compatibility issue.
> > > The goal is to make HBase easier for operators to upgrade, while it is
> > > also easier for developers to
> > > enhance, re-architect if necessary.
> > >
> > > The attached is a proposal we came up.  We'd like to start with two
> > phases:
> > >
> > > Phase 1: Compatibility between client applications and HBase clusters
> > > Phase 2: HBase cluster rolling upgrade within same major version
> > >
> > > Could you please review?
> > >
> > > Thanks,
> > > Jimmy
> > >
> >
>

It's the first bullet under Phase I.

Can you mention HBASE-5306 there ?

On Mon, Feb 13, 2012 at 1:18 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> What's the role of HBASE-5306 in your proposal ?
>
> Thanks
>
>
> On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>
>> I posted the proposal on wiki:
>>
>> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>>
>> Thanks,
>> Jimmy
>>
>> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>> > Can you post on wiki ?
>> >
>> > Attachment stripped.
>> >
>> > On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > > Hello,
>> > >
>> > > As HBase installation base is getting bigger, we are ready to work on
>> the
>> > > wire compatibility issue.
>> > > The goal is to make HBase easier for operators to upgrade, while it is
>> > > also easier for developers to
>> > > enhance, re-architect if necessary.
>> > >
>> > > The attached is a proposal we came up.  We'd like to start with two
>> > phases:
>> > >
>> > > Phase 1: Compatibility between client applications and HBase clusters
>> > > Phase 2: HBase cluster rolling upgrade within same major version
>> > >
>> > > Could you please review?
>> > >
>> > > Thanks,
>> > > Jimmy
>> > >
>> >
>>
>
>

Sure.  I will add that.  To me, HBASE-5306 is more about the RPC engine.
It can evolve in parallel since the current HBase RPC
can support PB already due to
HADOOP-7379<https://issues.apache.org/jira/browse/HADOOP-7379>

On Mon, Feb 13, 2012 at 1:23 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> It's the first bullet under Phase I.
>
> Can you mention HBASE-5306 there ?
>
> On Mon, Feb 13, 2012 at 1:18 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > What's the role of HBASE-5306 in your proposal ?
> >
> > Thanks
> >
> >
> > On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >
> >> I posted the proposal on wiki:
> >>
> >> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> >>
> >> Thanks,
> >> Jimmy
> >>
> >> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>
> >> > Can you post on wiki ?
> >> >
> >> > Attachment stripped.
> >> >
> >> > On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
> >> wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > > As HBase installation base is getting bigger, we are ready to work
> on
> >> the
> >> > > wire compatibility issue.
> >> > > The goal is to make HBase easier for operators to upgrade, while it
> is
> >> > > also easier for developers to
> >> > > enhance, re-architect if necessary.
> >> > >
> >> > > The attached is a proposal we came up.  We'd like to start with two
> >> > phases:
> >> > >
> >> > > Phase 1: Compatibility between client applications and HBase
> clusters
> >> > > Phase 2: HBase cluster rolling upgrade within same major version
> >> > >
> >> > > Could you please review?
> >> > >
> >> > > Thanks,
> >> > > Jimmy
> >> > >
> >> >
> >>
> >
> >
>

HADOOP-7379 was marked resolved for 0.23

Does this imply that hadoop 0.23 or above must be used ?
If so, we'd better state that on the wiki.

On Mon, Feb 13, 2012 at 2:04 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:

> Sure.  I will add that.  To me, HBASE-5306 is more about the RPC engine.
> It can evolve in parallel since the current HBase RPC
> can support PB already due to
> HADOOP-7379<https://issues.apache.org/jira/browse/HADOOP-7379>
>
> On Mon, Feb 13, 2012 at 1:23 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > It's the first bullet under Phase I.
> >
> > Can you mention HBASE-5306 there ?
> >
> > On Mon, Feb 13, 2012 at 1:18 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> > > What's the role of HBASE-5306 in your proposal ?
> > >
> > > Thanks
> > >
> > >
> > > On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> I posted the proposal on wiki:
> > >>
> > >> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> > >>
> > >> Thanks,
> > >> Jimmy
> > >>
> > >> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > >>
> > >> > Can you post on wiki ?
> > >> >
> > >> > Attachment stripped.
> > >> >
> > >> > On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
> > >> wrote:
> > >> >
> > >> > > Hello,
> > >> > >
> > >> > > As HBase installation base is getting bigger, we are ready to work
> > on
> > >> the
> > >> > > wire compatibility issue.
> > >> > > The goal is to make HBase easier for operators to upgrade, while
> it
> > is
> > >> > > also easier for developers to
> > >> > > enhance, re-architect if necessary.
> > >> > >
> > >> > > The attached is a proposal we came up.  We'd like to start with
> two
> > >> > phases:
> > >> > >
> > >> > > Phase 1: Compatibility between client applications and HBase
> > clusters
> > >> > > Phase 2: HBase cluster rolling upgrade within same major version
> > >> > >
> > >> > > Could you please review?
> > >> > >
> > >> > > Thanks,
> > >> > > Jimmy
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

We have a copy-paste fork of Hadoop IPC. So, it's easy enough to just
apply the same style patch to our fork of it.

On Mon, Feb 13, 2012 at 2:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> HADOOP-7379 was marked resolved for 0.23
>
> Does this imply that hadoop 0.23 or above must be used ?
> If so, we'd better state that on the wiki.
>
> On Mon, Feb 13, 2012 at 2:04 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>
>> Sure.  I will add that.  To me, HBASE-5306 is more about the RPC engine.
>> It can evolve in parallel since the current HBase RPC
>> can support PB already due to
>> HADOOP-7379<https://issues.apache.org/jira/browse/HADOOP-7379>
>>
>> On Mon, Feb 13, 2012 at 1:23 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>> > It's the first bullet under Phase I.
>> >
>> > Can you mention HBASE-5306 there ?
>> >
>> > On Mon, Feb 13, 2012 at 1:18 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> >
>> > > What's the role of HBASE-5306 in your proposal ?
>> > >
>> > > Thanks
>> > >
>> > >
>> > > On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > >> I posted the proposal on wiki:
>> > >>
>> > >> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>> > >>
>> > >> Thanks,
>> > >> Jimmy
>> > >>
>> > >> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > >>
>> > >> > Can you post on wiki ?
>> > >> >
>> > >> > Attachment stripped.
>> > >> >
>> > >> > On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
>> > >> wrote:
>> > >> >
>> > >> > > Hello,
>> > >> > >
>> > >> > > As HBase installation base is getting bigger, we are ready to work
>> > on
>> > >> the
>> > >> > > wire compatibility issue.
>> > >> > > The goal is to make HBase easier for operators to upgrade, while
>> it
>> > is
>> > >> > > also easier for developers to
>> > >> > > enhance, re-architect if necessary.
>> > >> > >
>> > >> > > The attached is a proposal we came up.  We'd like to start with
>> two
>> > >> > phases:
>> > >> > >
>> > >> > > Phase 1: Compatibility between client applications and HBase
>> > clusters
>> > >> > > Phase 2: HBase cluster rolling upgrade within same major version
>> > >> > >
>> > >> > > Could you please review?
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Jimmy
>> > >> > >
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>

--
Todd Lipcon
Software Engineer, Cloudera

I created HBASE-5394 for porting HADOOP-7379 to HBase

FYI

On Mon, Feb 13, 2012 at 2:22 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> We have a copy-paste fork of Hadoop IPC. So, it's easy enough to just
> apply the same style patch to our fork of it.
>
> On Mon, Feb 13, 2012 at 2:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > HADOOP-7379 was marked resolved for 0.23
> >
> > Does this imply that hadoop 0.23 or above must be used ?
> > If so, we'd better state that on the wiki.
> >
> > On Mon, Feb 13, 2012 at 2:04 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >
> >> Sure.  I will add that.  To me, HBASE-5306 is more about the RPC engine.
> >> It can evolve in parallel since the current HBase RPC
> >> can support PB already due to
> >> HADOOP-7379<https://issues.apache.org/jira/browse/HADOOP-7379>
> >>
> >> On Mon, Feb 13, 2012 at 1:23 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>
> >> > It's the first bullet under Phase I.
> >> >
> >> > Can you mention HBASE-5306 there ?
> >> >
> >> > On Mon, Feb 13, 2012 at 1:18 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > What's the role of HBASE-5306 in your proposal ?
> >> > >
> >> > > Thanks
> >> > >
> >> > >
> >> > > On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> >> > wrote:
> >> > >
> >> > >> I posted the proposal on wiki:
> >> > >>
> >> > >> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> >> > >>
> >> > >> Thanks,
> >> > >> Jimmy
> >> > >>
> >> > >> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]>
> wrote:
> >> > >>
> >> > >> > Can you post on wiki ?
> >> > >> >
> >> > >> > Attachment stripped.
> >> > >> >
> >> > >> > On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <
> [EMAIL PROTECTED]>
> >> > >> wrote:
> >> > >> >
> >> > >> > > Hello,
> >> > >> > >
> >> > >> > > As HBase installation base is getting bigger, we are ready to
> work
> >> > on
> >> > >> the
> >> > >> > > wire compatibility issue.
> >> > >> > > The goal is to make HBase easier for operators to upgrade,
> while
> >> it
> >> > is
> >> > >> > > also easier for developers to
> >> > >> > > enhance, re-architect if necessary.
> >> > >> > >
> >> > >> > > The attached is a proposal we came up.  We'd like to start with
> >> two
> >> > >> > phases:
> >> > >> > >
> >> > >> > > Phase 1: Compatibility between client applications and HBase
> >> > clusters
> >> > >> > > Phase 2: HBase cluster rolling upgrade within same major
> version
> >> > >> > >
> >> > >> > > Could you please review?
> >> > >> > >
> >> > >> > > Thanks,
> >> > >> > > Jimmy
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> >
> >>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Security considerations are mostly orthogonal to message encoding.

In the SecureRpcEngine there is a SASL negotiation at connection setup, and then HBase protocol data is transformed using the established context. That is the null transformation unless message integrity or confidentiality options are negotiated/required. The JRE's SASL support handles that. SASL is well defined and interoperable between versions. Otherwise we delegate to the HBase RPC code.

For ZooKeeper security, there is a new ZK protocol message type for the SASL authenticator. Unlike with HBase, the application protocol is not wrapped with a secure socket layer. So the authentication handshake is as cross version compatible as the rest of the ZK protocol, and the handshake basically tunnels SASL protocol messages, which are compatible cross version with respect to themselves. It was done this way due to how ZK architected pluggable authentication methods.

Best regards,

    - Andy
On Feb 14, 2012, at 5:02 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:

> I posted the proposal on wiki:
>
> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>
> Thanks,
> Jimmy
>
> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Can you post on wiki ?
>>
>> Attachment stripped.
>>
>> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>>
>>> Hello,
>>>
>>> As HBase installation base is getting bigger, we are ready to work on the
>>> wire compatibility issue.
>>> The goal is to make HBase easier for operators to upgrade, while it is
>>> also easier for developers to
>>> enhance, re-architect if necessary.
>>>
>>> The attached is a proposal we came up.  We'd like to start with two
>> phases:
>>>
>>> Phase 1: Compatibility between client applications and HBase clusters
>>> Phase 2: HBase cluster rolling upgrade within same major version
>>>
>>> Could you please review?
>>>
>>> Thanks,
>>> Jimmy
>>>
>>

On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
> I posted the proposal on wiki:
>
> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>
Looks great.  Looking at Phase 1, we're talking about a big bang where
we shut down cluster and come up on the new stuff w/ the new stuff
migrating old formats to new; e.g. the content in .META.  When are we
thinking we can take on the big bang?

Looking at phase 2, it looks like another big bang (should phase 1 and
phase 2 big bangs happen at same time)?

Should we swat team it?  Go away for a weekend and bang it out?

Can we make issues for each of the steps so we can discuss a bit of
detail on each of the bullets?

On:

- Should pluggable encodings (thrift/avro/pb/writable) be in scope?

I'd say no.

On:

- Should async IO servers and clients be in scope or not?

I'd say no for now.

On:

- What is the policy for existing versions (89, 90, 92, 94) -- do we
support them or require on major upgrade before they get this story?

I'm not sure how else to do it.  How do you make an old client work
against the new stuff?

On:

- Developers should be able to remove deprecated methods or arguments
to maintain flexibility, but can't do that within the compatibility
window. What should be our compatibility window? 2 years (roughly 4
major versions)?

Which compatibility window are you referring too?  Moving up on to
this platform will be the singularity across nothing can cross, right?

On:

- What is the ZK version interoperability story?

We need 3.4.x to support security.

On:

- What is the HDFS version interoperability story?

None or at least out of scope for this project.  Its another project?

On:

- Should architectural-level changes require a major version bump?

This is an interesting one.  For example, say we wanted to purge the
-ROOT- region.  Well, that'd be something an old client could not
tolerate.   So, you'd up the rpc version or figure how to purge -ROOT-
in a way that old clients can still work (then there is no need to
change rpc version -- to do a major version bump).

St.Ack

bq. Should we swat team it?

Or do this at the next Hackathon.

On Mon, Feb 13, 2012 at 4:09 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
> > I posted the proposal on wiki:
> >
> > http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> >
>
>
> Looks great.  Looking at Phase 1, we're talking about a big bang where
> we shut down cluster and come up on the new stuff w/ the new stuff
> migrating old formats to new; e.g. the content in .META.  When are we
> thinking we can take on the big bang?
>
> Looking at phase 2, it looks like another big bang (should phase 1 and
> phase 2 big bangs happen at same time)?
>
> Should we swat team it?  Go away for a weekend and bang it out?
>
> Can we make issues for each of the steps so we can discuss a bit of
> detail on each of the bullets?
>
> On:
>
> - Should pluggable encodings (thrift/avro/pb/writable) be in scope?
>
> I'd say no.
>
> On:
>
> - Should async IO servers and clients be in scope or not?
>
> I'd say no for now.
>
> On:
>
> - What is the policy for existing versions (89, 90, 92, 94) -- do we
> support them or require on major upgrade before they get this story?
>
> I'm not sure how else to do it.  How do you make an old client work
> against the new stuff?
>
> On:
>
> - Developers should be able to remove deprecated methods or arguments
> to maintain flexibility, but can't do that within the compatibility
> window. What should be our compatibility window? 2 years (roughly 4
> major versions)?
>
> Which compatibility window are you referring too?  Moving up on to
> this platform will be the singularity across nothing can cross, right?
>
> On:
>
> - What is the ZK version interoperability story?
>
> We need 3.4.x to support security.
>
> On:
>
> - What is the HDFS version interoperability story?
>
> None or at least out of scope for this project.  Its another project?
>
> On:
>
> - Should architectural-level changes require a major version bump?
>
> This is an interesting one.  For example, say we wanted to purge the
> -ROOT- region.  Well, that'd be something an old client could not
> tolerate.   So, you'd up the rpc version or figure how to purge -ROOT-
> in a way that old clients can still work (then there is no need to
> change rpc version -- to do a major version bump).
>
> St.Ack
>

It is a good idea to swat team it or hackathon.
We'd like to start with phase 1 at first, then phase 2.

The purpose is to reduce/eliminate the operation interruption.  We need to
start from somewhere.

Thanks,
Jimmy

On Mon, Feb 13, 2012 at 4:15 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> bq. Should we swat team it?
>
> Or do this at the next Hackathon.
>
> On Mon, Feb 13, 2012 at 4:09 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Mon, Feb 13, 2012 at 1:02 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> > > I posted the proposal on wiki:
> > >
> > > http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> > >
> >
> >
> > Looks great.  Looking at Phase 1, we're talking about a big bang where
> > we shut down cluster and come up on the new stuff w/ the new stuff
> > migrating old formats to new; e.g. the content in .META.  When are we
> > thinking we can take on the big bang?
> >
> > Looking at phase 2, it looks like another big bang (should phase 1 and
> > phase 2 big bangs happen at same time)?
> >
> > Should we swat team it?  Go away for a weekend and bang it out?
> >
> > Can we make issues for each of the steps so we can discuss a bit of
> > detail on each of the bullets?
> >
> > On:
> >
> > - Should pluggable encodings (thrift/avro/pb/writable) be in scope?
> >
> > I'd say no.
> >
> > On:
> >
> > - Should async IO servers and clients be in scope or not?
> >
> > I'd say no for now.
> >
> > On:
> >
> > - What is the policy for existing versions (89, 90, 92, 94) -- do we
> > support them or require on major upgrade before they get this story?
> >
> > I'm not sure how else to do it.  How do you make an old client work
> > against the new stuff?
> >
> > On:
> >
> > - Developers should be able to remove deprecated methods or arguments
> > to maintain flexibility, but can't do that within the compatibility
> > window. What should be our compatibility window? 2 years (roughly 4
> > major versions)?
> >
> > Which compatibility window are you referring too?  Moving up on to
> > this platform will be the singularity across nothing can cross, right?
> >
> > On:
> >
> > - What is the ZK version interoperability story?
> >
> > We need 3.4.x to support security.
> >
> > On:
> >
> > - What is the HDFS version interoperability story?
> >
> > None or at least out of scope for this project.  Its another project?
> >
> > On:
> >
> > - Should architectural-level changes require a major version bump?
> >
> > This is an interesting one.  For example, say we wanted to purge the
> > -ROOT- region.  Well, that'd be something an old client could not
> > tolerate.   So, you'd up the rpc version or figure how to purge -ROOT-
> > in a way that old clients can still work (then there is no need to
> > change rpc version -- to do a major version bump).
> >
> > St.Ack
> >
>

On Mon, Feb 13, 2012 at 4:58 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
> It is a good idea to swat team it or hackathon.
>
>

Little code has been written at hackathons in the past so it would
need to have a bit of a different format.

Or we just divvy up the work (I could do the migration of .META. and
-ROOT- to pb and of zk persistence to pb if wanted).
> We'd like to start with phase 1 at first, then phase 2.
>
> The purpose is to reduce/eliminate the operation interruption.  We need to
> start from somewhere.
>

Agree.

Phase 2 seems just as bad as a disruption as phase 1 or am I reading
it wrong; i.e. its another singularity that can't be crossed w/o a
restart of all servers.

I suppose I'm wondering if we think we can nail phase 1 and phase 2
within a single release cycle; then the two singularities are
experienced as one during a single upgrade.

St.Ack

On Mon, Feb 13, 2012 at 5:10 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Mon, Feb 13, 2012 at 4:58 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
> > It is a good idea to swat team it or hackathon.
> >
> >
>
> Little code has been written at hackathons in the past so it would
> need to have a bit of a different format.
>
> Or we just divvy up the work (I could do the migration of .META. and
> -ROOT- to pb and of zk persistence to pb if wanted).
>
>
That's great!
>
> > We'd like to start with phase 1 at first, then phase 2.
> >
> > The purpose is to reduce/eliminate the operation interruption.  We need
> to
> > start from somewhere.
> >
>
> Agree.
>
> Phase 2 seems just as bad as a disruption as phase 1 or am I reading
> it wrong; i.e. its another singularity that can't be crossed w/o a
> restart of all servers.
>
> I suppose I'm wondering if we think we can nail phase 1 and phase 2
> within a single release cycle; then the two singularities are
> experienced as one during a single upgrade.
>
>
Yes, if we can finish both in one release cycle.

Thanks,
Jimmy

> St.Ack
>

I think it would be good to have a swat team for discussing the open issues
and planning.  In the interest of discussing this while it is fresh in
everyone's mind, how about setting up a swat team at Cloudera Palo Alto (210
Portage Avenue) soon?

How about either:
1) Thursday Feb 16th @ 3:30 PM - 5:30 PM
2) Tuesday Feb 21st @ 1:00 PM - 3:00 PM
(we can end early or go a little longer if necessary)

Can people who want to come indicate their availability?

Thanks,
Greg

On Mon, Feb 13, 2012 at 5:21 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:

> On Mon, Feb 13, 2012 at 5:10 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Mon, Feb 13, 2012 at 4:58 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> > > It is a good idea to swat team it or hackathon.
> > >
> > >
> >
> > Little code has been written at hackathons in the past so it would
> > need to have a bit of a different format.
> >
> > Or we just divvy up the work (I could do the migration of .META. and
> > -ROOT- to pb and of zk persistence to pb if wanted).
> >
> >
> That's great!
>
>
> >
> > > We'd like to start with phase 1 at first, then phase 2.
> > >
> > > The purpose is to reduce/eliminate the operation interruption.  We need
> > to
> > > start from somewhere.
> > >
> >
> > Agree.
> >
> > Phase 2 seems just as bad as a disruption as phase 1 or am I reading
> > it wrong; i.e. its another singularity that can't be crossed w/o a
> > restart of all servers.
> >
> > I suppose I'm wondering if we think we can nail phase 1 and phase 2
> > within a single release cycle; then the two singularities are
> > experienced as one during a single upgrade.
> >
> >
> Yes, if we can finish both in one release cycle.
>
> Thanks,
> Jimmy
>
>
>
> > St.Ack
> >
>

On Tue, Feb 14, 2012 at 5:30 PM, Gregory Chanan <[EMAIL PROTECTED]> wrote:
> I think it would be good to have a swat team for discussing the open issues
> and planning.  In the interest of discussing this while it is fresh in
> everyone's mind, how about setting up a swat team at Cloudera Palo Alto (210
> Portage Avenue) soon?
>
> How about either:
> 1) Thursday Feb 16th @ 3:30 PM - 5:30 PM
> 2) Tuesday Feb 21st @ 1:00 PM - 3:00 PM

+1 for 2/21

-Todd

> On Mon, Feb 13, 2012 at 5:21 PM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>
>> On Mon, Feb 13, 2012 at 5:10 PM, Stack <[EMAIL PROTECTED]> wrote:
>>
>> > On Mon, Feb 13, 2012 at 4:58 PM, Jimmy Xiang <[EMAIL PROTECTED]>
>> wrote:
>> > > It is a good idea to swat team it or hackathon.
>> > >
>> > >
>> >
>> > Little code has been written at hackathons in the past so it would
>> > need to have a bit of a different format.
>> >
>> > Or we just divvy up the work (I could do the migration of .META. and
>> > -ROOT- to pb and of zk persistence to pb if wanted).
>> >
>> >
>> That's great!
>>
>>
>> >
>> > > We'd like to start with phase 1 at first, then phase 2.
>> > >
>> > > The purpose is to reduce/eliminate the operation interruption.  We need
>> > to
>> > > start from somewhere.
>> > >
>> >
>> > Agree.
>> >
>> > Phase 2 seems just as bad as a disruption as phase 1 or am I reading
>> > it wrong; i.e. its another singularity that can't be crossed w/o a
>> > restart of all servers.
>> >
>> > I suppose I'm wondering if we think we can nail phase 1 and phase 2
>> > within a single release cycle; then the two singularities are
>> > experienced as one during a single upgrade.
>> >
>> >
>> Yes, if we can finish both in one release cycle.
>>
>> Thanks,
>> Jimmy
>>
>>
>>
>> > St.Ack
>> >
>>

--
Todd Lipcon
Software Engineer, Cloudera

Count me in for 21st.

On Tue, Feb 14, 2012 at 5:57 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Tue, Feb 14, 2012 at 5:30 PM, Gregory Chanan <[EMAIL PROTECTED]>
> wrote:
> > I think it would be good to have a swat team for discussing the open
> issues
> > and planning.  In the interest of discussing this while it is fresh in
> > everyone's mind, how about setting up a swat team at Cloudera Palo Alto
> (210
> > Portage Avenue) soon?
> >
> > How about either:
> > 1) Thursday Feb 16th @ 3:30 PM - 5:30 PM
> > 2) Tuesday Feb 21st @ 1:00 PM - 3:00 PM
>
> +1 for 2/21
>
> -Todd
>
> > On Mon, Feb 13, 2012 at 5:21 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >
> >> On Mon, Feb 13, 2012 at 5:10 PM, Stack <[EMAIL PROTECTED]> wrote:
> >>
> >> > On Mon, Feb 13, 2012 at 4:58 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> >> wrote:
> >> > > It is a good idea to swat team it or hackathon.
> >> > >
> >> > >
> >> >
> >> > Little code has been written at hackathons in the past so it would
> >> > need to have a bit of a different format.
> >> >
> >> > Or we just divvy up the work (I could do the migration of .META. and
> >> > -ROOT- to pb and of zk persistence to pb if wanted).
> >> >
> >> >
> >> That's great!
> >>
> >>
> >> >
> >> > > We'd like to start with phase 1 at first, then phase 2.
> >> > >
> >> > > The purpose is to reduce/eliminate the operation interruption.  We
> need
> >> > to
> >> > > start from somewhere.
> >> > >
> >> >
> >> > Agree.
> >> >
> >> > Phase 2 seems just as bad as a disruption as phase 1 or am I reading
> >> > it wrong; i.e. its another singularity that can't be crossed w/o a
> >> > restart of all servers.
> >> >
> >> > I suppose I'm wondering if we think we can nail phase 1 and phase 2
> >> > within a single release cycle; then the two singularities are
> >> > experienced as one during a single upgrade.
> >> >
> >> >
> >> Yes, if we can finish both in one release cycle.
> >>
> >> Thanks,
> >> Jimmy
> >>
> >>
> >>
> >> > St.Ack
> >> >
> >>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

+1 on 2/21.

________________________________
 From: Ted Yu <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Cc: Xiaoyu Zhi <[EMAIL PROTECTED]>
Sent: Wednesday, February 15, 2012 12:09 PM
Subject: Re: HBase wire compatibility
 
Count me in for 21st.

On Tue, Feb 14, 2012 at 5:57 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Tue, Feb 14, 2012 at 5:30 PM, Gregory Chanan <[EMAIL PROTECTED]>
> wrote:
> > I think it would be good to have a swat team for discussing the open
> issues
> > and planning.  In the interest of discussing this while it is fresh in
> > everyone's mind, how about setting up a swat team at Cloudera Palo Alto
> (210
> > Portage Avenue) soon?
> >
> > How about either:
> > 1) Thursday Feb 16th @ 3:30 PM - 5:30 PM
> > 2) Tuesday Feb 21st @ 1:00 PM - 3:00 PM
>
> +1 for 2/21
>
> -Todd
>
> > On Mon, Feb 13, 2012 at 5:21 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >
> >> On Mon, Feb 13, 2012 at 5:10 PM, Stack <[EMAIL PROTECTED]> wrote:
> >>
> >> > On Mon, Feb 13, 2012 at 4:58 PM, Jimmy Xiang <[EMAIL PROTECTED]>
> >> wrote:
> >> > > It is a good idea to swat team it or hackathon.
> >> > >
> >> > >
> >> >
> >> > Little code has been written at hackathons in the past so it would
> >> > need to have a bit of a different format.
> >> >
> >> > Or we just divvy up the work (I could do the migration of .META. and
> >> > -ROOT- to pb and of zk persistence to pb if wanted).
> >> >
> >> >
> >> That's great!
> >>
> >>
> >> >
> >> > > We'd like to start with phase 1 at first, then phase 2.
> >> > >
> >> > > The purpose is to reduce/eliminate the operation interruption.  We
> need
> >> > to
> >> > > start from somewhere.
> >> > >
> >> >
> >> > Agree.
> >> >
> >> > Phase 2 seems just as bad as a disruption as phase 1 or am I reading
> >> > it wrong; i.e. its another singularity that can't be crossed w/o a
> >> > restart of all servers.
> >> >
> >> > I suppose I'm wondering if we think we can nail phase 1 and phase 2
> >> > within a single release cycle; then the two singularities are
> >> > experienced as one during a single upgrade.
> >> >
> >> >
> >> Yes, if we can finish both in one release cycle.
> >>
> >> Thanks,
> >> Jimmy
> >>
> >>
> >>
> >> > St.Ack
> >> >
> >>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

I found that my flight back to US arrives at SFO @ 12:55pm on 21st.

If there is no way to move the meetup a little later, I will have to pass
this one.

Cheers

On Wed, Feb 15, 2012 at 12:09 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Count me in for 21st.
>
>
> On Tue, Feb 14, 2012 at 5:57 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> On Tue, Feb 14, 2012 at 5:30 PM, Gregory Chanan <[EMAIL PROTECTED]>
>> wrote:
>> > I think it would be good to have a swat team for discussing the open
>> issues
>> > and planning.  In the interest of discussing this while it is fresh in
>> > everyone's mind, how about setting up a swat team at Cloudera Palo Alto
>> (210
>> > Portage Avenue) soon?
>> >
>> > How about either:
>> > 1) Thursday Feb 16th @ 3:30 PM - 5:30 PM
>> > 2) Tuesday Feb 21st @ 1:00 PM - 3:00 PM
>>
>> +1 for 2/21
>>
>> -Todd
>>
>> > On Mon, Feb 13, 2012 at 5:21 PM, Jimmy Xiang <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> On Mon, Feb 13, 2012 at 5:10 PM, Stack <[EMAIL PROTECTED]> wrote:
>> >>
>> >> > On Mon, Feb 13, 2012 at 4:58 PM, Jimmy Xiang <[EMAIL PROTECTED]>
>> >> wrote:
>> >> > > It is a good idea to swat team it or hackathon.
>> >> > >
>> >> > >
>> >> >
>> >> > Little code has been written at hackathons in the past so it would
>> >> > need to have a bit of a different format.
>> >> >
>> >> > Or we just divvy up the work (I could do the migration of .META. and
>> >> > -ROOT- to pb and of zk persistence to pb if wanted).
>> >> >
>> >> >
>> >> That's great!
>> >>
>> >>
>> >> >
>> >> > > We'd like to start with phase 1 at first, then phase 2.
>> >> > >
>> >> > > The purpose is to reduce/eliminate the operation interruption.  We
>> need
>> >> > to
>> >> > > start from somewhere.
>> >> > >
>> >> >
>> >> > Agree.
>> >> >
>> >> > Phase 2 seems just as bad as a disruption as phase 1 or am I reading
>> >> > it wrong; i.e. its another singularity that can't be crossed w/o a
>> >> > restart of all servers.
>> >> >
>> >> > I suppose I'm wondering if we think we can nail phase 1 and phase 2
>> >> > within a single release cycle; then the two singularities are
>> >> > experienced as one during a single upgrade.
>> >> >
>> >> >
>> >> Yes, if we can finish both in one release cycle.
>> >>
>> >> Thanks,
>> >> Jimmy
>> >>
>> >>
>> >>
>> >> > St.Ack
>> >> >
>> >>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>

On Tue, Feb 14, 2012 at 5:30 PM, Gregory Chanan <[EMAIL PROTECTED]> wrote:
> I think it would be good to have a swat team for discussing the open issues
> and planning.  In the interest of discussing this while it is fresh in
> everyone's mind, how about setting up a swat team at Cloudera Palo Alto (210
> Portage Avenue) soon?
>
> How about either:
> 1) Thursday Feb 16th @ 3:30 PM - 5:30 PM
> 2) Tuesday Feb 21st @ 1:00 PM - 3:00 PM
> (we can end early or go a little longer if necessary)
>

2/21 works for me.
St.Ack

On Tue, Feb 14, 2012 at 5:30 PM, Gregory Chanan <[EMAIL PROTECTED]> wrote:
>> How about either:
...
> 2) Tuesday Feb 21st @ 1:00 PM - 3:00 PM
> (we can end early or go a little longer if necessary)
If anyone needs a ride from and to SF, write me offline.
St.Ack

Good writeup, Jimmy (was away for a few days due to an event in my family)
 
Some quick questions - Has there been any thoughts on the plan to use HBASE-5394. Are we going to make the hbase protocols (like HRegionInterface) protobuf aware?

In Hadoop, I have seen the following:
1. In HDFS, the protocol definitions are not changed (like org.apache.hadoop.hdfs.protocol.ClientProtocol). Instead there are translators that are defined that implement the mapping of protobuf datastructures to application-level datastructures and vice versa (for example, have a look at ClientNamenodeProtocolTranslatorPB and ClientNamenodeProtocolServerSideTranslatorPB in the package org.apache.hadoop.hdfs.protocolPB).
2. In Yarn (MRV2), all protocol definitions are written in PB
 
Since the base RPC still uses writables for payload encoding, a translation happens when the protobuf objects are sent/received (as an example look at org.apache.hadoop.ipc.ProtobufRpcEngine; classes RpcRequestWritable and RpcResponseWritable).
 
What does the HBase community think about the above?
On Feb 13, 2012, at 1:02 PM, Jimmy Xiang wrote:

> I posted the proposal on wiki:
>
> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>
> Thanks,
> Jimmy
>
> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> Can you post on wiki ?
>>
>> Attachment stripped.
>>
>> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>>
>>> Hello,
>>>
>>> As HBase installation base is getting bigger, we are ready to work on the
>>> wire compatibility issue.
>>> The goal is to make HBase easier for operators to upgrade, while it is
>>> also easier for developers to
>>> enhance, re-architect if necessary.
>>>
>>> The attached is a proposal we came up.  We'd like to start with two
>> phases:
>>>
>>> Phase 1: Compatibility between client applications and HBase clusters
>>> Phase 2: HBase cluster rolling upgrade within same major version
>>>
>>> Could you please review?
>>>
>>> Thanks,
>>> Jimmy
>>>
>>

Hi Devaraj,

IMO, the "protocol translator" interface introduced in HDFS has been
more pain than it's worth, given pluggability seems like a non-goal
for us. I think it's easier to just tie ourselves to protobufs. In
essence the client library (e.g. HTable) acts as the translator
between the user types and the wire types (protobuf).

That said, we should be absolutely sure that we don't expose protobufs
to the *client API*

-Todd

On Thu, Feb 16, 2012 at 12:30 PM, Devaraj Das <[EMAIL PROTECTED]> wrote:
> Good writeup, Jimmy (was away for a few days due to an event in my family)
>
> Some quick questions - Has there been any thoughts on the plan to use HBASE-5394. Are we going to make the hbase protocols (like HRegionInterface) protobuf aware?
>
> In Hadoop, I have seen the following:
> 1. In HDFS, the protocol definitions are not changed (like org.apache.hadoop.hdfs.protocol.ClientProtocol). Instead there are translators that are defined that implement the mapping of protobuf datastructures to application-level datastructures and vice versa (for example, have a look at ClientNamenodeProtocolTranslatorPB and ClientNamenodeProtocolServerSideTranslatorPB in the package org.apache.hadoop.hdfs.protocolPB).
> 2. In Yarn (MRV2), all protocol definitions are written in PB
>
> Since the base RPC still uses writables for payload encoding, a translation happens when the protobuf objects are sent/received (as an example look at org.apache.hadoop.ipc.ProtobufRpcEngine; classes RpcRequestWritable and RpcResponseWritable).
>
> What does the HBase community think about the above?
>
>
> On Feb 13, 2012, at 1:02 PM, Jimmy Xiang wrote:
>
>> I posted the proposal on wiki:
>>
>> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>>
>> Thanks,
>> Jimmy
>>
>> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>>> Can you post on wiki ?
>>>
>>> Attachment stripped.
>>>
>>> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hello,
>>>>
>>>> As HBase installation base is getting bigger, we are ready to work on the
>>>> wire compatibility issue.
>>>> The goal is to make HBase easier for operators to upgrade, while it is
>>>> also easier for developers to
>>>> enhance, re-architect if necessary.
>>>>
>>>> The attached is a proposal we came up.  We'd like to start with two
>>> phases:
>>>>
>>>> Phase 1: Compatibility between client applications and HBase clusters
>>>> Phase 2: HBase cluster rolling upgrade within same major version
>>>>
>>>> Could you please review?
>>>>
>>>> Thanks,
>>>> Jimmy
>>>>
>>>
>

--
Todd Lipcon
Software Engineer, Cloudera

Hi Devaraj,

HBASE-5394 is resolved.  Now HBaseRPC can support both Writable and PB.

There will be a swat team meeting for this next week.  Greg will send out
the meeting
time and place soon.  You, and all developers/users interested in this are
welcomed to join, discuss and come up an agreement.

Thanks,
Jimmy
On Thu, Feb 16, 2012 at 12:48 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hi Devaraj,
>
> IMO, the "protocol translator" interface introduced in HDFS has been
> more pain than it's worth, given pluggability seems like a non-goal
> for us. I think it's easier to just tie ourselves to protobufs. In
> essence the client library (e.g. HTable) acts as the translator
> between the user types and the wire types (protobuf).
>
> That said, we should be absolutely sure that we don't expose protobufs
> to the *client API*
>
> -Todd
>
> On Thu, Feb 16, 2012 at 12:30 PM, Devaraj Das <[EMAIL PROTECTED]>
> wrote:
> > Good writeup, Jimmy (was away for a few days due to an event in my
> family)
> >
> > Some quick questions - Has there been any thoughts on the plan to use
> HBASE-5394. Are we going to make the hbase protocols (like
> HRegionInterface) protobuf aware?
> >
> > In Hadoop, I have seen the following:
> > 1. In HDFS, the protocol definitions are not changed (like
> org.apache.hadoop.hdfs.protocol.ClientProtocol). Instead there are
> translators that are defined that implement the mapping of protobuf
> datastructures to application-level datastructures and vice versa (for
> example, have a look at ClientNamenodeProtocolTranslatorPB and
> ClientNamenodeProtocolServerSideTranslatorPB in the package
> org.apache.hadoop.hdfs.protocolPB).
> > 2. In Yarn (MRV2), all protocol definitions are written in PB
> >
> > Since the base RPC still uses writables for payload encoding, a
> translation happens when the protobuf objects are sent/received (as an
> example look at org.apache.hadoop.ipc.ProtobufRpcEngine; classes
> RpcRequestWritable and RpcResponseWritable).
> >
> > What does the HBase community think about the above?
> >
> >
> > On Feb 13, 2012, at 1:02 PM, Jimmy Xiang wrote:
> >
> >> I posted the proposal on wiki:
> >>
> >> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> >>
> >> Thanks,
> >> Jimmy
> >>
> >> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>
> >>> Can you post on wiki ?
> >>>
> >>> Attachment stripped.
> >>>
> >>> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> As HBase installation base is getting bigger, we are ready to work on
> the
> >>>> wire compatibility issue.
> >>>> The goal is to make HBase easier for operators to upgrade, while it is
> >>>> also easier for developers to
> >>>> enhance, re-architect if necessary.
> >>>>
> >>>> The attached is a proposal we came up.  We'd like to start with two
> >>> phases:
> >>>>
> >>>> Phase 1: Compatibility between client applications and HBase clusters
> >>>> Phase 2: HBase cluster rolling upgrade within same major version
> >>>>
> >>>> Could you please review?
> >>>>
> >>>> Thanks,
> >>>> Jimmy
> >>>>
> >>>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

On Thu, Feb 16, 2012 at 12:48 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

That said, we should be absolutely sure that we don't expose protobufs
> to the *client API*
As a pure user & outsider, I have to share that right now we use protobufs
extensively all around HBase and constant conversion between byte[],
ImmutableBytesWritable and ByteString seems like a lot of overhead
(especially with larger objects).  While exposing ByteString etc. as the
only interface might not be very user friendly, exposing it as an advanced
interface would be very useful.  Or at a minimum, it would be nice that if
we wanted to get access to the lower level pb objects, that modifications
to HTable and/or supporting classes wouldn't be super complicated.

just my .02,
Jacques

On Thu, Feb 16, 2012 at 2:27 PM, Jacques <[EMAIL PROTECTED]> wrote:
> Or at a minimum, it would be nice that if
> we wanted to get access to the lower level pb objects, that modifications
> to HTable and/or supporting classes wouldn't be super complicated.

It's less a matter of complexity, and more a matter of not wanting to
expose the implementation details as part of the API. It really
restricts us when we do this -- for example, KeyValue is used in the
underlying storage all the way up through the client API, which means
it's verify difficult for us to do something like switch to an
off-heap storage for cell data, for example.

That said, the request for an easy way to build convenience APIs with
low numbers of copies makes sense

-Todd
--
Todd Lipcon
Software Engineer, Cloudera

Fair enough.  That's why I mentioned ByteString more than anything else.
 If the new RPC will always convert client api/application values into
ByteStrings and my application is always storing protobuf keys and protobuf
objects, it'd be nice if I could just hand you a ByteString as the value
for each of these rather than converting them back to byte[] and then
having you convert them again.

thanks,
Jacques

On Thu, Feb 16, 2012 at 2:33 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Thu, Feb 16, 2012 at 2:27 PM, Jacques <[EMAIL PROTECTED]> wrote:
> > Or at a minimum, it would be nice that if
> > we wanted to get access to the lower level pb objects, that modifications
> > to HTable and/or supporting classes wouldn't be super complicated.
>
> It's less a matter of complexity, and more a matter of not wanting to
> expose the implementation details as part of the API. It really
> restricts us when we do this -- for example, KeyValue is used in the
> underlying storage all the way up through the client API, which means
> it's verify difficult for us to do something like switch to an
> off-heap storage for cell data, for example.
>
> That said, the request for an easy way to build convenience APIs with
> low numbers of copies makes sense
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Given the +1s, let's do the meetup at:

Tuesday Feb 21st @ 1:00 PM - 3:00 PM
Cloudera Palo Alto
210 Portage Ave,
Palo Alto, CA 94306
Let me know if you have any questions about getting here, etc.

@Ted: I prefer 1 pm because people already agreed to it and it gives us
more time if people want to stay later for more discussion or coding.  You
are certainly welcome to join us late.

Greg

On Thu, Feb 16, 2012 at 2:41 PM, Jacques <[EMAIL PROTECTED]> wrote:

> Fair enough.  That's why I mentioned ByteString more than anything else.
>  If the new RPC will always convert client api/application values into
> ByteStrings and my application is always storing protobuf keys and protobuf
> objects, it'd be nice if I could just hand you a ByteString as the value
> for each of these rather than converting them back to byte[] and then
> having you convert them again.
>
> thanks,
> Jacques
>
> On Thu, Feb 16, 2012 at 2:33 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Feb 16, 2012 at 2:27 PM, Jacques <[EMAIL PROTECTED]> wrote:
> > > Or at a minimum, it would be nice that if
> > > we wanted to get access to the lower level pb objects, that
> modifications
> > > to HTable and/or supporting classes wouldn't be super complicated.
> >
> > It's less a matter of complexity, and more a matter of not wanting to
> > expose the implementation details as part of the API. It really
> > restricts us when we do this -- for example, KeyValue is used in the
> > underlying storage all the way up through the client API, which means
> > it's verify difficult for us to do something like switch to an
> > off-heap storage for cell data, for example.
> >
> > That said, the request for an easy way to build convenience APIs with
> > low numbers of copies makes sense
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>

Will this allow for hbase clients in other languages?  It seems that if pb are being used then any
language pb supports could have a first class client and not have to use a separate (and not super
maintained) thrift server. You would want to keep the client to be light though if it were to be
ported to other languages.

IMHO it seems that if we are going to the work of redoing the client communications we should be
considering other languages.  It seems like having first class clients in various languages could
only increase hbase adoption which would be a good thing :-)  I would really like to see hbase be
more usable from other languages besides java.

~Jeff

On 2/13/2012 12:01 PM, Jimmy Xiang wrote:
> Hello,
>
> As HBase installation base is getting bigger, we are ready to work on the wire compatibility issue.
> The goal is to make HBase easier for operators to upgrade, while it is also easier for developers to
> enhance, re-architect if necessary.
>
> The attached is a proposal we came up.  We'd like to start with two phases:
>
> Phase 1: Compatibility between client applications and HBase clusters
> Phase 2: HBase cluster rolling upgrade within same major version
>
> Could you please review?
>
> Thanks,
> Jimmy

--
Jeff Whiting
Qualtrics Senior Software Engineer
[EMAIL PROTECTED]

On Thu, Feb 16, 2012 at 3:04 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> Will this allow for hbase clients in other languages?  It seems that if pb
> are being used then any language pb supports could have a first class client
> and not have to use a separate (and not super maintained) thrift server. You
> would want to keep the client to be light though if it were to be ported to
> other languages.
>
> IMHO it seems that if we are going to the work of redoing the client
> communications we should be considering other languages.  It seems like
> having first class clients in various languages could only increase hbase
> adoption which would be a good thing :-)  I would really like to see hbase
> be more usable from other languages besides java.

The issue is that HBase's client is necessarily not thin. It requires
a lot of knowledge of HBase itself -- so certainly moving to PB would
get us one step closer, but it would still be quite a bit of work to
write a new client in another language. Certainly if someone comes
along with one, that would be nice, but I don't think we should take
it upon the project (yet) to maintain any other language clients.

-Todd

>
>
> On 2/13/2012 12:01 PM, Jimmy Xiang wrote:
>>
>> Hello,
>>
>> As HBase installation base is getting bigger, we are ready to work on the
>> wire compatibility issue.
>> The goal is to make HBase easier for operators to upgrade, while it is
>> also easier for developers to
>> enhance, re-architect if necessary.
>>
>> The attached is a proposal we came up.  We'd like to start with two
>> phases:
>>
>> Phase 1: Compatibility between client applications and HBase clusters
>> Phase 2: HBase cluster rolling upgrade within same major version
>>
>> Could you please review?
>>
>> Thanks,
>> Jimmy
>
>
> --
> Jeff Whiting
> Qualtrics Senior Software Engineer
> [EMAIL PROTECTED]
>

--
Todd Lipcon
Software Engineer, Cloudera

One of my colleagues have developed a extensive functional hbase-client in
C++ and we are in the process of open-sourcing it. It uses the Thrift
Interface to interact with the reigonservers.

-dhruba
On Thu, Feb 16, 2012 at 3:09 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> On Thu, Feb 16, 2012 at 3:04 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> > Will this allow for hbase clients in other languages?  It seems that if
> pb
> > are being used then any language pb supports could have a first class
> client
> > and not have to use a separate (and not super maintained) thrift server.
> You
> > would want to keep the client to be light though if it were to be ported
> to
> > other languages.
> >
> > IMHO it seems that if we are going to the work of redoing the client
> > communications we should be considering other languages.  It seems like
> > having first class clients in various languages could only increase hbase
> > adoption which would be a good thing :-)  I would really like to see
> hbase
> > be more usable from other languages besides java.
>
> The issue is that HBase's client is necessarily not thin. It requires
> a lot of knowledge of HBase itself -- so certainly moving to PB would
> get us one step closer, but it would still be quite a bit of work to
> write a new client in another language. Certainly if someone comes
> along with one, that would be nice, but I don't think we should take
> it upon the project (yet) to maintain any other language clients.
>
> -Todd
>
> >
> >
> > On 2/13/2012 12:01 PM, Jimmy Xiang wrote:
> >>
> >> Hello,
> >>
> >> As HBase installation base is getting bigger, we are ready to work on
> the
> >> wire compatibility issue.
> >> The goal is to make HBase easier for operators to upgrade, while it is
> >> also easier for developers to
> >> enhance, re-architect if necessary.
> >>
> >> The attached is a proposal we came up.  We'd like to start with two
> >> phases:
> >>
> >> Phase 1: Compatibility between client applications and HBase clusters
> >> Phase 2: HBase cluster rolling upgrade within same major version
> >>
> >> Could you please review?
> >>
> >> Thanks,
> >> Jimmy
> >
> >
> > --
> > Jeff Whiting
> > Qualtrics Senior Software Engineer
> > [EMAIL PROTECTED]
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

--
Subscribe to my posts at http://www.facebook.com/dhruba

It seems like the only heavy part of the client would be the zookeeper interactions (forgive my
ignorance if I'm wrong).  Other than zookeeper only a basic understanding of regions need to be
understood.  So if the zookeeper interactions could be removed and pushed somewhere else in the
stack that could make the client much thinner.

I understand not maintaining multiple clients nor do I think the project should maintain another
client right now.  However now seems like the time to plan for new clients in other languages.  When
will be the next time the client / server protocol is changed?  I'm guessing most people would say
hopefully never again.  IMHO since you are redoing the communication why not improve the protocol to
allow for a leaner the client.  A leaner client would be more likely to work across major hbase
changes, would be easier to maintain, would hide implementation details and could have less
dependencies.  One of the reasons the client doesn't do well across major changes is because of how
heavy it is. Even if the client is never implemented in another language a thinner client would seem
to be an improvement.

What I'm suggesting may not be possible but it seems like it is worth keeping in mind through the
design and implementation process.

~Jeff

On 2/16/2012 4:09 PM, Todd Lipcon wrote:
> On Thu, Feb 16, 2012 at 3:04 PM, Jeff Whiting<[EMAIL PROTECTED]>  wrote:
>> Will this allow for hbase clients in other languages?  It seems that if pb
>> are being used then any language pb supports could have a first class client
>> and not have to use a separate (and not super maintained) thrift server. You
>> would want to keep the client to be light though if it were to be ported to
>> other languages.
>>
>> IMHO it seems that if we are going to the work of redoing the client
>> communications we should be considering other languages.  It seems like
>> having first class clients in various languages could only increase hbase
>> adoption which would be a good thing :-)  I would really like to see hbase
>> be more usable from other languages besides java.
> The issue is that HBase's client is necessarily not thin. It requires
> a lot of knowledge of HBase itself -- so certainly moving to PB would
> get us one step closer, but it would still be quite a bit of work to
> write a new client in another language. Certainly if someone comes
> along with one, that would be nice, but I don't think we should take
> it upon the project (yet) to maintain any other language clients.
>
> -Todd
>
>>
>> On 2/13/2012 12:01 PM, Jimmy Xiang wrote:
>>> Hello,
>>>
>>> As HBase installation base is getting bigger, we are ready to work on the
>>> wire compatibility issue.
>>> The goal is to make HBase easier for operators to upgrade, while it is
>>> also easier for developers to
>>> enhance, re-architect if necessary.
>>>
>>> The attached is a proposal we came up.  We'd like to start with two
>>> phases:
>>>
>>> Phase 1: Compatibility between client applications and HBase clusters
>>> Phase 2: HBase cluster rolling upgrade within same major version
>>>
>>> Could you please review?
>>>
>>> Thanks,
>>> Jimmy
>>
>> --
>> Jeff Whiting
>> Qualtrics Senior Software Engineer
>> [EMAIL PROTECTED]
>>
>
>

--
Jeff Whiting
Qualtrics Senior Software Engineer
[EMAIL PROTECTED]

On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> What I'm suggesting may not be possible but it seems like it is worth
> keeping in mind through the design and implementation process.
>

There is already proof that what you suggest is possible.  asynchbase
is a 'lighter', 'incomplete' but
complete-enough-for-a-particular-purpose client that does much of what
you talk of except the bit about being in another language.

St.Ack

While working on keeping bw compatibility for
https://issues.apache.org/jira/browse/HBASE-5371, I realized that it might
be a good idea to include the co-processor interfaces into the scope as
well. Although they are marked as advanced API's, and even if you wrap them
under some java class, there will be need to access the functionality from
clients, and they should be treated the same way as 3 and 4. Just something
to consider until Thursday, I'll try to be there as well.

Thanks,
Enis

On Thu, Feb 16, 2012 at 4:54 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> > What I'm suggesting may not be possible but it seems like it is worth
> > keeping in mind through the design and implementation process.
> >
>
> There is already proof that what you suggest is possible.  asynchbase
> is a 'lighter', 'incomplete' but
> complete-enough-for-a-particular-purpose client that does much of what
> you talk of except the bit about being in another language.
>
> St.Ack
>

Enis,

You mean until Tuesday (the 21st), right?

Greg

On Fri, Feb 17, 2012 at 5:30 PM, Enis Söztutar <[EMAIL PROTECTED]> wrote:

> While working on keeping bw compatibility for
> https://issues.apache.org/jira/browse/HBASE-5371, I realized that it might
> be a good idea to include the co-processor interfaces into the scope as
> well. Although they are marked as advanced API's, and even if you wrap them
> under some java class, there will be need to access the functionality from
> clients, and they should be treated the same way as 3 and 4. Just something
> to consider until Thursday, I'll try to be there as well.
>
> Thanks,
> Enis
>
> On Thu, Feb 16, 2012 at 4:54 PM, Stack <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting <[EMAIL PROTECTED]>
> wrote:
> > > What I'm suggesting may not be possible but it seems like it is worth
> > > keeping in mind through the design and implementation process.
> > >
> >
> > There is already proof that what you suggest is possible.  asynchbase
> > is a 'lighter', 'incomplete' but
> > complete-enough-for-a-particular-purpose client that does much of what
> > you talk of except the bit about being in another language.
> >
> > St.Ack
> >
>

On Fri, Feb 17, 2012 at 5:42 PM, Gregory Chanan <[EMAIL PROTECTED]>wrote:

> Enis,
>
> You mean until Tuesday (the 21st), right?
>
>
Ops, my bad. Will be there today.

Enis
> Greg
>
> On Fri, Feb 17, 2012 at 5:30 PM, Enis Söztutar <[EMAIL PROTECTED]> wrote:
>
> > While working on keeping bw compatibility for
> > https://issues.apache.org/jira/browse/HBASE-5371, I realized that it
> might
> > be a good idea to include the co-processor interfaces into the scope as
> > well. Although they are marked as advanced API's, and even if you wrap
> them
> > under some java class, there will be need to access the functionality
> from
> > clients, and they should be treated the same way as 3 and 4. Just
> something
> > to consider until Thursday, I'll try to be there as well.
> >
> > Thanks,
> > Enis
> >
> > On Thu, Feb 16, 2012 at 4:54 PM, Stack <[EMAIL PROTECTED]> wrote:
> >
> > > On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting <[EMAIL PROTECTED]>
> > wrote:
> > > > What I'm suggesting may not be possible but it seems like it is worth
> > > > keeping in mind through the design and implementation process.
> > > >
> > >
> > > There is already proof that what you suggest is possible.  asynchbase
> > > is a 'lighter', 'incomplete' but
> > > complete-enough-for-a-particular-purpose client that does much of what
> > > you talk of except the bit about being in another language.
> > >
> > > St.Ack
> > >
> >
>

> While working on keeping bw compatibility for
> https://issues.apache.org/jira/browse/HBASE-5371, I realized that it might
> be a good idea to include the co-processor interfaces into the scope as

> well

+1

Wire compatibility considerations should extend to dynamic RPC protocols/endpoints and extensibility of their custom protocols. Whatever choices are made, they will impact how coprocessor implementors can maintain cross version compatibility themselves.

I think the general direction of using protobufs is good, most issues with cross-versioning of protocols and extensibility are handled. And it's reasonable for an implementor of an advanced API to deal with more implementation particulars than others (users).

One thing I do worry about is some RPC layer change that breaks dynamic endpoints. That can't happen.

Best regards,
    - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)

----- Original Message -----
> From: Enis Söztutar <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Friday, February 17, 2012 5:30 PM
> Subject: Re: HBase wire compatibility
>
> While working on keeping bw compatibility for
> https://issues.apache.org/jira/browse/HBASE-5371, I realized that it might
> be a good idea to include the co-processor interfaces into the scope as
> well. Although they are marked as advanced API's, and even if you wrap them
> under some java class, there will be need to access the functionality from
> clients, and they should be treated the same way as 3 and 4. Just something
> to consider until Thursday, I'll try to be there as well.
>
> Thanks,
> Enis
>
> On Thu, Feb 16, 2012 at 4:54 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>>  On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting <[EMAIL PROTECTED]>
> wrote:
>>  > What I'm suggesting may not be possible but it seems like it is
> worth
>>  > keeping in mind through the design and implementation process.
>>  >
>>
>>  There is already proof that what you suggest is possible.  asynchbase
>>  is a 'lighter', 'incomplete' but
>>  complete-enough-for-a-particular-purpose client that does much of what
>>  you talk of except the bit about being in another language.
>>
>>  St.Ack
>>
>

On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
> It seems like the only heavy part of the client would be the zookeeper
> interactions (forgive my ignorance if I'm wrong).

ZooKeeper interactions are extremely simple for a client, that's not
where the heavy part is.  All a client needs to do with ZooKeeper is
to find where the -ROOT- region is, period.  In the client I wrote,
asynchbase, I don't even maintain an open connection to ZooKeeper,
because 99.99% of the time it's unnecessary.

> Other than zookeeper only
> a basic understanding of regions need to be understood.  So if the zookeeper
> interactions could be removed and pushed somewhere else in the stack that
> could make the client much thinner.

Using line count (per "wc -l") as a rough approximation of code
complexity, here's a break down of asynchbase.  For a total of 11k
lines the big chunks of code are:
ZooKeeper code: 360 lines (not actually big but I included it for comparison)
Code for handling NoSuchRegionException: 500 lines
Helper code to deal with byte arrays: 500 lines
Helper code to deal with HBase RPC serialization: 700 lines
Code to batch RPCs: 800 lines
Low-level socket code, and wire serialization/deserialization: 800 lines
Code to open, manage, close scanners: 1000 lines
Code for looking up and caching regions: 1000 lines

> hopefully never again.  IMHO since you are redoing the communication why not
> improve the protocol to allow for a leaner the client.  A leaner client
> would be more likely to work across major hbase changes, would be easier to
> maintain, would hide implementation details and could have less
> dependencies.

Yes a leaner client would be better.  But the reason the client is fat
is because Bigtable's design pushed a lot of logic down to the clients
in order to be able to make RPC routing decisions there, and relieve
the tablet servers from having to do it.  When you start to have tens
of thousands of clients talking to a cluster, like Google does, it
makes sense to push this work down to the many clients, rather than
have the fewer TabletServers do it and re-route packets (adding extra
hops etc).  The overall system is more efficient this way.

Leaner clients are better, but unfortunately lean clients are often
dumb, so it's hard to find a good tradeoff between simplicity and
efficiency.

> One of the reasons the client doesn't do well across major
> changes is because of how heavy it is. Even if the client is never
> implemented in another language a thinner client would seem to be an
> improvement.

Having maintained an HBase client written from scratch for about 2
years now, I can tell you that the only things I had to fix across
HBase release were wire-level serialization breakages.  The heavy
logic of the client has remained mostly unchanged since the days of
HBase 0.20.

--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com

Thanks for the explanation.  I enjoyed hearing your perspective.

~Jeff

On 2/22/2012 1:20 PM, tsuna wrote:
> On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting<[EMAIL PROTECTED]>  wrote:
>> It seems like the only heavy part of the client would be the zookeeper
>> interactions (forgive my ignorance if I'm wrong).
> ZooKeeper interactions are extremely simple for a client, that's not
> where the heavy part is.  All a client needs to do with ZooKeeper is
> to find where the -ROOT- region is, period.  In the client I wrote,
> asynchbase, I don't even maintain an open connection to ZooKeeper,
> because 99.99% of the time it's unnecessary.
>
>>   Other than zookeeper only
>> a basic understanding of regions need to be understood.  So if the zookeeper
>> interactions could be removed and pushed somewhere else in the stack that
>> could make the client much thinner.
> Using line count (per "wc -l") as a rough approximation of code
> complexity, here's a break down of asynchbase.  For a total of 11k
> lines the big chunks of code are:
> ZooKeeper code: 360 lines (not actually big but I included it for comparison)
> Code for handling NoSuchRegionException: 500 lines
> Helper code to deal with byte arrays: 500 lines
> Helper code to deal with HBase RPC serialization: 700 lines
> Code to batch RPCs: 800 lines
> Low-level socket code, and wire serialization/deserialization: 800 lines
> Code to open, manage, close scanners: 1000 lines
> Code for looking up and caching regions: 1000 lines
>
>> hopefully never again.  IMHO since you are redoing the communication why not
>> improve the protocol to allow for a leaner the client.  A leaner client
>> would be more likely to work across major hbase changes, would be easier to
>> maintain, would hide implementation details and could have less
>> dependencies.
> Yes a leaner client would be better.  But the reason the client is fat
> is because Bigtable's design pushed a lot of logic down to the clients
> in order to be able to make RPC routing decisions there, and relieve
> the tablet servers from having to do it.  When you start to have tens
> of thousands of clients talking to a cluster, like Google does, it
> makes sense to push this work down to the many clients, rather than
> have the fewer TabletServers do it and re-route packets (adding extra
> hops etc).  The overall system is more efficient this way.
>
> Leaner clients are better, but unfortunately lean clients are often
> dumb, so it's hard to find a good tradeoff between simplicity and
> efficiency.
>
>>   One of the reasons the client doesn't do well across major
>> changes is because of how heavy it is. Even if the client is never
>> implemented in another language a thinner client would seem to be an
>> improvement.
> Having maintained an HBase client written from scratch for about 2
> years now, I can tell you that the only things I had to fix across
> HBase release were wire-level serialization breakages.  The heavy
> logic of the client has remained mostly unchanged since the days of
> HBase 0.20.
>

--
Jeff Whiting
Qualtrics Senior Software Engineer
[EMAIL PROTECTED]

On Wed, Feb 22, 2012 at 12:20 PM, tsuna <[EMAIL PROTECTED]> wrote:
> On Thu, Feb 16, 2012 at 3:55 PM, Jeff Whiting <[EMAIL PROTECTED]> wrote:
>> It seems like the only heavy part of the client would be the zookeeper
>> interactions (forgive my ignorance if I'm wrong).
>
> ZooKeeper interactions are extremely simple for a client, that's not
> where the heavy part is.  All a client needs to do with ZooKeeper is
> to find where the -ROOT- region is, period.  In the client I wrote,
> asynchbase, I don't even maintain an open connection to ZooKeeper,
> because 99.99% of the time it's unnecessary.
>

Our N is working on doing this for the hbase client:
https://issues.apache.org/jira/browse/HBASE-5399
St.Ack