Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - HBase wire compatibility


Copy link to this message
-
Re: HBase wire compatibility
Jimmy Xiang 2012-02-16, 20:56
Hi Devaraj,

HBASE-5394 is resolved.  Now HBaseRPC can support both Writable and PB.

There will be a swat team meeting for this next week.  Greg will send out
the meeting
time and place soon.  You, and all developers/users interested in this are
welcomed to join, discuss and come up an agreement.

Thanks,
Jimmy
On Thu, Feb 16, 2012 at 12:48 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hi Devaraj,
>
> IMO, the "protocol translator" interface introduced in HDFS has been
> more pain than it's worth, given pluggability seems like a non-goal
> for us. I think it's easier to just tie ourselves to protobufs. In
> essence the client library (e.g. HTable) acts as the translator
> between the user types and the wire types (protobuf).
>
> That said, we should be absolutely sure that we don't expose protobufs
> to the *client API*
>
> -Todd
>
> On Thu, Feb 16, 2012 at 12:30 PM, Devaraj Das <[EMAIL PROTECTED]>
> wrote:
> > Good writeup, Jimmy (was away for a few days due to an event in my
> family)
> >
> > Some quick questions - Has there been any thoughts on the plan to use
> HBASE-5394. Are we going to make the hbase protocols (like
> HRegionInterface) protobuf aware?
> >
> > In Hadoop, I have seen the following:
> > 1. In HDFS, the protocol definitions are not changed (like
> org.apache.hadoop.hdfs.protocol.ClientProtocol). Instead there are
> translators that are defined that implement the mapping of protobuf
> datastructures to application-level datastructures and vice versa (for
> example, have a look at ClientNamenodeProtocolTranslatorPB and
> ClientNamenodeProtocolServerSideTranslatorPB in the package
> org.apache.hadoop.hdfs.protocolPB).
> > 2. In Yarn (MRV2), all protocol definitions are written in PB
> >
> > Since the base RPC still uses writables for payload encoding, a
> translation happens when the protobuf objects are sent/received (as an
> example look at org.apache.hadoop.ipc.ProtobufRpcEngine; classes
> RpcRequestWritable and RpcResponseWritable).
> >
> > What does the HBase community think about the above?
> >
> >
> > On Feb 13, 2012, at 1:02 PM, Jimmy Xiang wrote:
> >
> >> I posted the proposal on wiki:
> >>
> >> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> >>
> >> Thanks,
> >> Jimmy
> >>
> >> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>
> >>> Can you post on wiki ?
> >>>
> >>> Attachment stripped.
> >>>
> >>> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> As HBase installation base is getting bigger, we are ready to work on
> the
> >>>> wire compatibility issue.
> >>>> The goal is to make HBase easier for operators to upgrade, while it is
> >>>> also easier for developers to
> >>>> enhance, re-architect if necessary.
> >>>>
> >>>> The attached is a proposal we came up.  We'd like to start with two
> >>> phases:
> >>>>
> >>>> Phase 1: Compatibility between client applications and HBase clusters
> >>>> Phase 2: HBase cluster rolling upgrade within same major version
> >>>>
> >>>> Could you please review?
> >>>>
> >>>> Thanks,
> >>>> Jimmy
> >>>>
> >>>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>