Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> modular build and pluggable rpc


Copy link to this message
-
Re: modular build and pluggable rpc
+1 on maven modules. That will simplify the native code build/integration
that I'm working on for HBASE-1316.

-Joey
On May 27, 2011 6:15 PM, "Ryan Rawson" <[EMAIL PROTECTED]> wrote:
> The build modules are fine, I just wanted to voice my opinions on avro
> vs thrift. I dont think we should spend a lot of time attempting to
> build a avro vs thrift thing, we should plan to eventually move to
> thrift as our RPC serialization. I also concur with Todd, our server
> side code has had a lot of work and it isnt half bad now :-)
>
> +1 to maven modules, they are pretty cool
>
> On Fri, May 27, 2011 at 2:38 PM, Andrew Purtell <[EMAIL PROTECTED]>
wrote:
>> I don't disagree with any of this but the fact is we have compile time
differences if going against secure Hadoop 0.20 or non-secure Hadoop 0.20.
>>
>> So either we decide to punt on integration with secure Hadoop 0.20 or we
deal with the compile time differences. If dealing with them, we can do it
by reflection, which is brittle and can be difficult to understand and
debug, and someone would have to do this work; or we can wholesale replace
RPC with something based on Thrift, and someone would have to do the work;
or we take the pluggable RPC changes that Gary has already developed and
modularize the build, which Eric has already volunteered to do.
>>
>>  - Andy
>>
>> --- On Fri, 5/27/11, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>
>>> From: Todd Lipcon <[EMAIL PROTECTED]>
>>> Subject: Re: modular build and pluggable rpc
>>> To: [EMAIL PROTECTED]
>>> Cc: [EMAIL PROTECTED]
>>> Date: Friday, May 27, 2011, 1:30 PM
>>> Agreed - I'm all for Thrift.
>>>
>>> Though, I actually, contrary to Ryan, think that the
>>> existing HBaseRPC
>>> handler/client code is pretty good -- better than the
>>> equivalents from
>>> Thrift Java.
>>>
>>> We could start by using Thrift serialization on our
>>> existing transport
>>> -- then maybe work towards contributing it upstream to the
>>> Thrift
>>> project. HDFS folks are potentially interested in doing
>>> that as well.
>>>
>>> -Todd
>>>
>>> On Fri, May 27, 2011 at 1:10 PM, Ryan Rawson <[EMAIL PROTECTED]>
>>> wrote:
>>> > I'm -1 on avro as a RPC format.  Thrift is the way to
>>> go, any of the
>>> > advantages of smaller serialization of avro is lost by
>>> the sheer
>>> > complexity of avro and therefore the potential bugs.
>>> >
>>> > I understand the desire to have a pluggable RPC
>>> engine, but it feels
>>> > like the better approach would be to adopt a unified
>>> RPC and just be
>>> > done with it.  I had a look at the HsHa mechanism in
>>> thrift and it is
>>> > very good, it in fact matches our 'handler' approach -
>>> async
>>> > recieving/sending of data, but single threaded for
>>> processing a
>>> > message.
>>> >
>>> > -ryan
>>> >
>>> > On Fri, May 27, 2011 at 1:00 PM, Andrew Purtell <[EMAIL PROTECTED]>
>>> wrote:
>>> >> Also needing, perhaps later, consideration:
>>> >>
>>> >> - HDFS-347 or not
>>> >>
>>> >>  - Lucene embedding for hbase-search, though as a
>>> coprocessor this is already pretty much handled if we have
>>> platform support (therefore a platform module) for a HDFS
>>> that can do local read shortcutting and block placement
>>> requests
>>> >>
>>> >> - HFile v1 versus v2
>>> >>
>>> >> Making decoupled development at several downstream
>>> sites manageable, with a home upstream for all the work,
>>> while simultaneously providing clean migration paths for
>>> users, basically.
>>> >>
>>> >> --- On Fri, 5/27/11, Andrew Purtell <[EMAIL PROTECTED]>
>>> wrote:
>>> >>
>>> >>> From: Andrew Purtell <[EMAIL PROTECTED]>
>>> >>> Subject: modular build and pluggable rpc
>>> >>> To: [EMAIL PROTECTED]
>>> >>> Date: Friday, May 27, 2011, 12:49 PM
>>> >>> From IRC:
>>> >>>
>>> >>> apurtell    i propose we take the build
>>> modular as early as possible to deal with multiple platform
>>> targets
>>> >>> apurtell    secure vs nonsecure
>>> >>> apurtell    0.20 vs 0.22 vs trunk
>>> >>> apurtell    i understand the maintenence
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB