Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> Re: HBASE-2182


Copy link to this message
-
Re: HBASE-2182
Whoops dev got dropped; adding it back on.

Since everything is already a protobuf, to me it doesn't really make sense
to keep the hadoop serialization overhead too.  In addition the netty
protocol allows for zero copy, which would be pretty tough to implement
with the older rpc format.

On Sat, Jun 30, 2012 at 2:31 PM, ryan rawson <[EMAIL PROTECTED]> wrote:

> But isn't worth losing compatibility? If you have compatibility you can do
> each side separately
>
> Sent from your iPhone
>
> On Jun 30, 2012, at 1:57 PM, Elliott Clark <[EMAIL PROTECTED]> wrote:
>
> That protocol was chosen because netty provides a ready built
> implementation of the encoder and the decoder.  Looking at the code for the
> two, they are pretty easy to work on if needed.
>
> On Sat, Jun 30, 2012 at 1:24 AM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
>
>> Hey Elliott,
>>
>> I saw this comment in the bug:
>>
>> "All communication takes place through a wrapped protocol buffer protocol:
>> [32 bit length field, protobuf data]"
>>
>> HBase RPC already includes a framing protocol, that is every message
>> is prefixed with a 4-byte size of the following data.  Eg:
>> HBaseServer.java:1157 is where the request buffer is allocated server
>> side, and HBaseServer.java:353 for the response to the client.
>>
>> I'm interested in better HBase network clients, and I have a few ideas
>> of how to approach the problem.  Netty is obviously the ideal way to
>> approach it I think.
>>
>> What is your intent and ability to work on this line of code?
>>
>> -ryan
>>
>>
>>
>>
>>
>> On Fri, Jun 29, 2012 at 4:42 PM, Elliott Clark <[EMAIL PROTECTED]>
>> wrote:
>> > I just posted a pretty early skeleton(
>> > https://issues.apache.org/jira/browse/HBASE-2182) on what I think a
>> netty
>> > based hbase client/server could look like.
>> >
>> > Pros:
>> >
>> >   - Faster
>> >      - Giraph got a 3x perf improvement by droppping hadoop rpc
>> >      - Asynhbase trounces our client when JD benchmarked them
>> >   - Could encourage things to be a little more modular if everything
>> isn't
>> >   hanging directly off of HRegionServer
>> >   - Netty is better about thread usage than hadoop rpc server.
>> >   - Pretty easy to define an rpc protocol after all of the work on
>> >   protobuf (Thanks everyone)
>> >   - Decoupling the rpc server library from the hadoop library could
>> allow
>> >   us to rev the server code easier.
>> >   - The filter model is very easy to work with.
>> >      - Security can be just a single filter.
>> >      - Logging can ba another
>> >      - Stats can be another.
>> >
>> > Cons:
>> >
>> >   - Netty and non apache rpc server's don't play well togther.  They
>> might
>> >   be able to but I haven't gotten there yet.
>> >   - Complexity
>> >      - Two different servers in the src
>> >      - Confusing users who don't know which to pick
>> >   - Non-blocking could make the client a harder to write.
>> >
>> >
>> > I'm really just trying to gauge what people think of the direction and
>> if
>> > it's still something that is wanted.  The code is a loooooong way from
>> even
>> > being a tech demo, and I'm not a netty expert, so suggestions would be
>> > welcomed.
>> >
>> > Thoughts ? Are people interested in this? Should I push this to my
>> github
>> > so other can help ?
>>
>
>
+
Elliott Clark 2012-06-29, 23:42
+
Todd Lipcon 2012-06-30, 00:04
+
Ryan Rawson 2012-06-30, 08:27
+
N Keywal 2012-06-30, 11:50
+
Andrew Purtell 2012-06-30, 00:34
+
Elliott Clark 2012-06-30, 00:59
+
Andrew Purtell 2012-06-30, 01:10
+
Elliott Clark 2012-06-30, 00:21