Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # dev >> DISCUSS : HFile V3 proposal for tags in 0.96

ramkrishna vasudevan 2013-07-18, 17:14
Ted Yu 2013-07-18, 17:23
Jimmy Xiang 2013-07-18, 17:55
ramkrishna vasudevan 2013-07-19, 04:29
Ted Yu 2013-07-19, 04:40
ramkrishna vasudevan 2013-07-19, 04:57
Ted Yu 2013-07-19, 05:00
Stack 2013-07-19, 05:12
Jean-Marc Spaggiari 2013-07-19, 11:02
ramkrishna vasudevan 2013-07-19, 11:11
Jean-Marc Spaggiari 2013-07-19, 11:23
Anoop John 2013-07-19, 11:51
ramkrishna vasudevan 2013-07-19, 12:00
Jean-Marc Spaggiari 2013-07-19, 13:09
Ted Yu 2013-07-19, 14:18
Anoop John 2013-07-19, 15:12
ramkrishna vasudevan 2013-07-19, 15:05
Andrew Purtell 2013-07-19, 16:27
Ted Yu 2013-07-19, 16:32
Anoop John 2013-07-19, 17:13
Ted Yu 2013-07-19, 17:35
Elliott Clark 2013-07-19, 17:52
Andrew Purtell 2013-07-19, 18:01
Elliott Clark 2013-07-19, 21:02
Andrew Purtell 2013-07-19, 22:34
Stack 2013-07-19, 23:31
Copy link to this message
Re: DISCUSS : HFile V3 proposal for tags in 0.96
>>What sort of changes do you foresee necessary in core to support
cellcodecs?  Between rpc and hfilev3?

Agree that codec provides the maximum flexibility to define how the cells
are serailized/deserialized be it be the RPC or WAL.

The RPC or the WAL codec just deals with the KVs in a sequential way.
 Every KV that comes in is just persisted and read back in the same way.
Also in the RPC and WAL the mvcc(memstoreTS) does not have any significance.

When we come into the HFile layer we end up having seekable reads and it is
not always sequential.  Also the HFile reader/writer and the block encoder
are tightly coupled with the KeyValue structure.
The datablock encoder have a seeker inside them.  We need something similar
to that in side the HFile codecs.

Also the current way of how the HFile datablock works

Current write path
=================HFileWriter->append(kv) > form Hfileblock byte buffer>Encoders read the
bytebuffer-> Encode(based on algo) per kv into new bytebuffer-> The new
bytebuffer is persisted.

The read path creates an EncodedScanner for encoded buffers or a plain
Scanner for no encoding.
Every algo has its own Seeker interfaces.
Introduction of Codec in this read path would need usage of a Decoder which
should be seekable.  Also we will need different Decoder impl for each of
the algo.

Now introducing codec in the current write path would mean that

HFileWriter->Codec.encode(kv) -> form hfileblock byte buffer > Use
codec.decoder to read the bytebuffer -> Encode(based on algo) per kv into
new bytebuffer -> The new byte buffer is persisted.

with the current seeker interfaces it is difficult to perform rewind(),
next(), seekToKeyInBlock() with codec (on current code).
To support the changes that may be needed to support cell codecs in HFile
we basically need to change/modify the existing interfaces.
If the current api for Encoding/Decoding of the codecs are seen it
basically has a advance() and current().  If we do not want to modify the
basic Decoder interface then we would be creating a
And all the seeker related apis would go into them.  Based on every codec
that we write we would need to ensure that the HFileEncoder/Decoder has its
algo of the datablockEncoders like KVcodecFastdiff, KVCodecDiffKey etc.

So when ever the codec changes we need to create our own algos that knows
how to deal with the codec way of decoding.
This also means that when NONE encoding algo is used, the codec should
create a seeker that would help to deal with plain byte buffers.
This would mean that the current BufferedDataEncoder would not directly
help us as they are tightly coupled with KV strucutre.

*Thanks to Anoop for his advice on the above design when i was stuck with
the way how to proceed with codecs.*

Now if we think we don't change any of the interfaces but just introduce V3
that would work with Codecs, then to illustrate the problem, how to
next(), blockSeek(), readKeyValueLen() in ScannerV3 (just taking an eg)

We would just be doing
 private final void readKeyValueLen() {
+      try {
+        // TODO : Can we specify the max value here
+        decoder.mark(Integer.MAX_VALUE);
+        decoder.advance();
+        currKV = decoder.current();
+        decoder.reset();
+      } catch (IOException e) {
+        throw new RuntimeException("Error while reading the keyvalue len",
+      }
+    }

But here the thing is for doing seekBefore() and seekTO() we would need to
everytime read a full KV using the decoder and then take the necessary
whether to go back or move to the next one.  Basically this will be a
costly operation considering the criticality of the read path.

Hence we would need some seekers as mentioned above who will work hand in
hand with the codec that is specified.  Hence the above changes would mean
changing HFileV3 with codec would affect major portion of the existing code
and hence we avoid it from doing for now.

PrefixTreecodec has it own version of Reversible scanner that knows to deal
with the prefixtree structure.  We may need something similar for HFiles
with some seekable()/rewindable() type of apis.  And these seekable()
things should be customised per codec.

Also we should also think on introducing codecs that would be easier to
work with Cells.  That is another area that would need to be checked before
we finalise the apis for HFile related codecs.  Basically Cells do not deal
with keys as we do now in seekTo(), seekBefore().  This we can have in
another discussion.

On Sat, Jul 20, 2013 at 5:01 AM, Stack <[EMAIL PROTECTED]> wrote:

Andrew Purtell 2013-07-22, 17:23
Jean-Marc Spaggiari 2013-07-23, 22:43
Andrew Purtell 2013-07-24, 17:33
ramkrishna vasudevan 2013-07-25, 18:09
Andrew Purtell 2013-07-19, 16:48
Ted Yu 2013-07-24, 17:30
Andrew Purtell 2013-07-19, 16:25
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB