Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> HBase wire compatibility


Copy link to this message
-
Re: HBase wire compatibility
Hi Devaraj,

HBASE-5394 is resolved.  Now HBaseRPC can support both Writable and PB.

There will be a swat team meeting for this next week.  Greg will send out
the meeting
time and place soon.  You, and all developers/users interested in this are
welcomed to join, discuss and come up an agreement.

Thanks,
Jimmy
On Thu, Feb 16, 2012 at 12:48 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hi Devaraj,
>
> IMO, the "protocol translator" interface introduced in HDFS has been
> more pain than it's worth, given pluggability seems like a non-goal
> for us. I think it's easier to just tie ourselves to protobufs. In
> essence the client library (e.g. HTable) acts as the translator
> between the user types and the wire types (protobuf).
>
> That said, we should be absolutely sure that we don't expose protobufs
> to the *client API*
>
> -Todd
>
> On Thu, Feb 16, 2012 at 12:30 PM, Devaraj Das <[EMAIL PROTECTED]>
> wrote:
> > Good writeup, Jimmy (was away for a few days due to an event in my
> family)
> >
> > Some quick questions - Has there been any thoughts on the plan to use
> HBASE-5394. Are we going to make the hbase protocols (like
> HRegionInterface) protobuf aware?
> >
> > In Hadoop, I have seen the following:
> > 1. In HDFS, the protocol definitions are not changed (like
> org.apache.hadoop.hdfs.protocol.ClientProtocol). Instead there are
> translators that are defined that implement the mapping of protobuf
> datastructures to application-level datastructures and vice versa (for
> example, have a look at ClientNamenodeProtocolTranslatorPB and
> ClientNamenodeProtocolServerSideTranslatorPB in the package
> org.apache.hadoop.hdfs.protocolPB).
> > 2. In Yarn (MRV2), all protocol definitions are written in PB
> >
> > Since the base RPC still uses writables for payload encoding, a
> translation happens when the protobuf objects are sent/received (as an
> example look at org.apache.hadoop.ipc.ProtobufRpcEngine; classes
> RpcRequestWritable and RpcResponseWritable).
> >
> > What does the HBase community think about the above?
> >
> >
> > On Feb 13, 2012, at 1:02 PM, Jimmy Xiang wrote:
> >
> >> I posted the proposal on wiki:
> >>
> >> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
> >>
> >> Thanks,
> >> Jimmy
> >>
> >> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>
> >>> Can you post on wiki ?
> >>>
> >>> Attachment stripped.
> >>>
> >>> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]>
> wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> As HBase installation base is getting bigger, we are ready to work on
> the
> >>>> wire compatibility issue.
> >>>> The goal is to make HBase easier for operators to upgrade, while it is
> >>>> also easier for developers to
> >>>> enhance, re-architect if necessary.
> >>>>
> >>>> The attached is a proposal we came up.  We'd like to start with two
> >>> phases:
> >>>>
> >>>> Phase 1: Compatibility between client applications and HBase clusters
> >>>> Phase 2: HBase cluster rolling upgrade within same major version
> >>>>
> >>>> Could you please review?
> >>>>
> >>>> Thanks,
> >>>> Jimmy
> >>>>
> >>>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB