Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> HBase wire compatibility


Copy link to this message
-
Re: HBase wire compatibility
Hi Devaraj,

IMO, the "protocol translator" interface introduced in HDFS has been
more pain than it's worth, given pluggability seems like a non-goal
for us. I think it's easier to just tie ourselves to protobufs. In
essence the client library (e.g. HTable) acts as the translator
between the user types and the wire types (protobuf).

That said, we should be absolutely sure that we don't expose protobufs
to the *client API*

-Todd

On Thu, Feb 16, 2012 at 12:30 PM, Devaraj Das <[EMAIL PROTECTED]> wrote:
> Good writeup, Jimmy (was away for a few days due to an event in my family)
>
> Some quick questions - Has there been any thoughts on the plan to use HBASE-5394. Are we going to make the hbase protocols (like HRegionInterface) protobuf aware?
>
> In Hadoop, I have seen the following:
> 1. In HDFS, the protocol definitions are not changed (like org.apache.hadoop.hdfs.protocol.ClientProtocol). Instead there are translators that are defined that implement the mapping of protobuf datastructures to application-level datastructures and vice versa (for example, have a look at ClientNamenodeProtocolTranslatorPB and ClientNamenodeProtocolServerSideTranslatorPB in the package org.apache.hadoop.hdfs.protocolPB).
> 2. In Yarn (MRV2), all protocol definitions are written in PB
>
> Since the base RPC still uses writables for payload encoding, a translation happens when the protobuf objects are sent/received (as an example look at org.apache.hadoop.ipc.ProtobufRpcEngine; classes RpcRequestWritable and RpcResponseWritable).
>
> What does the HBase community think about the above?
>
>
> On Feb 13, 2012, at 1:02 PM, Jimmy Xiang wrote:
>
>> I posted the proposal on wiki:
>>
>> http://wiki.apache.org/hadoop/Hbase/HBaseWireCompatibility
>>
>> Thanks,
>> Jimmy
>>
>> On Mon, Feb 13, 2012 at 11:03 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>>> Can you post on wiki ?
>>>
>>> Attachment stripped.
>>>
>>> On Mon, Feb 13, 2012 at 11:01 AM, Jimmy Xiang <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hello,
>>>>
>>>> As HBase installation base is getting bigger, we are ready to work on the
>>>> wire compatibility issue.
>>>> The goal is to make HBase easier for operators to upgrade, while it is
>>>> also easier for developers to
>>>> enhance, re-architect if necessary.
>>>>
>>>> The attached is a proposal we came up.  We'd like to start with two
>>> phases:
>>>>
>>>> Phase 1: Compatibility between client applications and HBase clusters
>>>> Phase 2: HBase cluster rolling upgrade within same major version
>>>>
>>>> Could you please review?
>>>>
>>>> Thanks,
>>>> Jimmy
>>>>
>>>
>

--
Todd Lipcon
Software Engineer, Cloudera