Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> RPC KeyValue encoding

Copy link to this message
Re: RPC KeyValue encoding

If we make the KeyValue wire format flexible enough I think we'll be able
to tackle the KV as an interface work later.  Just throwing out some ideas

We could have a byte at the front of each KV serialization format that
gives various options in each bit e.g.
Omits Rows / Omits Family / Omits Qualifier / Omits Timestamp / Omits Value
/ plus some extra bytes for compression options and extensions.  Then we
just need to define where the KV gets its field if it is omitted, e.g. from
the previous KV in the RPC that had that field filled in.  We sort of have
this with the optional fields already, although I don't recall exactly how
protobuf handles those (we'd probably have to do some small restructuring);
what's new is defining what it means when a field is omitted.

There's some overhead with the above for small KVs, so you could also go
coarser grain, e.g. the Get request/response could have a similar options
byte like:
All Share Same Row / All Share Same Family / ... / and one of the bits
could turn on the finer grain options above (per KeyValue).

The advantage of this is that all we'd have to get right in 0.96.0 is the
deserialization.  The serialization could just send without any of the
options turned on.  And we could experiment later with each specific RPC
call what the best options to use are, as well as what storage to actually
use client/server side, which you discuss.


On Sun, Sep 2, 2012 at 9:04 AM, Ted Yu <[EMAIL PROTECTED]> wrote:

> Thanks for the update, Matt.
> w.r.t. Cell class, since it is so fundamental, should it reside in org.
> apache.hadoop.hbase namespace as KeyValue class does ?
> For CellAppender, is compile() equivalent to flushing ?
> Looking forward to your publishing on the reviewboard.
> On Sat, Sep 1, 2012 at 11:29 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:
> > RPC encoding would be really nice since there is sometimes significant
> wire
> > traffic that could be reduced many-fold.  I have a particular table that
> i
> > scan and stream to a gzipped output file on S3, and i've noticed that
> while
> > the app server's network input is 100Mbps, the gzipped output can be
> 2Mbps!
> >
> > Finishing the PrefixTree has been slow because I've saved a couple tricky
> > issues to the end and am light on time.  i'll try to put it on
> reviewboard
> > monday despite a known bug.  It is built with some of the ideas you
> mention
> > in mind, Lars.  Take a look at the
> > Cell<
> >
> https://github.com/hotpads/hbase/blob/prefix-tree/hbase-common/src/main/java/org/apache/hadoop/hbase/cell/Cell.java
> > >
> >  and CellAppender<
> >
> https://github.com/hotpads/hbase/blob/prefix-tree/hbase-common/src/main/java/org/apache/hadoop/hbase/cell/appender/CellAppender.java
> > >
> > classes
> > and their comments.  The idea with the CellAppender is to stream cells
> into
> > it and periodically compile()/flush() into a byte[] which can be saved to
> > an HFile or (eventually) sent over the wire.  For example, in
> > HRegion.get(..), the CellAppender would replace the "ArrayList<KeyValue>
> > results" collection.
> >
> > After introducing the Cell interface, the trick to extending the encoded
> > cells up the HBase stack will be to reduce the reliance on stand-alone
> > KeyValues.  We'll want things like the Filters and KeyValueHeap to be
> able
> > to operate on reused Cells without materializing them into full
> KeyValues.
> >  That means that something like StoreFileScanner.peek() will not work
> > because the scanner cannot maintain the state of the currrent and next
> > Cells at the same time.  See
> > CellCollator<
> >
> https://github.com/hotpads/hbase/blob/prefix-tree/hbase-common/src/main/java/org/apache/hadoop/hbase/cell/collator/CellCollator.java
> > >
> > for
> > a possible replacement for KeyValueHeap.  The good news is that this can
> be
> > done in stages without major disruptions to the code base.
> >
> > Looking at PtDataBlockEncoderSeeker<
> >
> https://github.com/hotpads/hbase/blob/prefix-tree/hbase-prefix-tree/src/main/java/org/apache/hbase/codec/prefixtree/PtDataBlockEncoderSeeker.java