Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Question about HFile seeking


+
Varun Sharma 2013-05-16, 20:56
+
Varun Sharma 2013-05-16, 21:03
+
Stack 2013-05-16, 21:30
+
Stack 2013-05-16, 21:30
+
Varun Sharma 2013-05-16, 21:42
+
Varun Sharma 2013-05-16, 21:55
+
Varun Sharma 2013-05-16, 22:26
+
lars hofhansl 2013-05-17, 06:32
Copy link to this message
-
Re: Question about HFile seeking
On Thu, May 16, 2013 at 3:26 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> Referring to your comment above again
>
> "If you doing a prefix scan w/ row1c, we should be starting the scan at
> row1c, not row1 (or more correctly at the row that starts the block we
> believe has a row1c row in it...)"
>
> I am trying to understand how you could seek right across to the block
> containing "row1c" using the HFile Index. If the index is just built on
> HFile keys and there is no demarcation b/w rows and col(s), you would hit
> the block for "row1,col1". After that you would either need a way to skip
> right across to "row1c" after you find that this is not the row you are
> looking for or you will have to simply keep scanning and discarding
> sequentially until you get "row1c". If you have to keep scanning and
> discarding, then that is probably suboptimal. But if there is a way to skip
> right across from "row1,col1" to "row1c", then thats great, though I wonder
> how that would be implemented.
>
There is no demarcation but the serialized row key in the index has the
lengths of each of the row key constituent parts embedded.

Here is where we write out an entry:

http://hbase.apache.org/xref/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.html#292

The entry is a KeyValue instances.  See how we pass the key and value
component separately into here:

http://hbase.apache.org/xref/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.html#325

The key part is serialized in a particular format.  It is described here:

http://hbase.apache.org/xref/org/apache/hadoop/hbase/KeyValue.html#58

(or see http://hbase.apache.org/book.html#keyvalue, where it is better
described... easier to read).

Notice how the serialization has row length, column family length,
qualifier length, etc.

While in KV, see how it has comparators that compare byte arrays that have
serialized keys in them first by row, then by column family, etc: i.e. by
each of the component parts in turn:
http://hbase.apache.org/xref/org/apache/hadoop/hbase/KeyValue.html#2515

Here is where we save off the key if it is first in current block:

http://hbase.apache.org/xref/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.html#353

It gets added to index when we figure we have a block full of data (search
checkBlockBoundary in HFileWriterV2).

So keys in index are full-on keys.

At seek time, we eventually get down to an index lookup.  See here in hfile
package:

http://hbase.apache.org/xref/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.html#176

Notice how there is a comparator and a byte array w/ a key in it.  The
comparator we read from the file.  It was comparator used writing the file.
 If you wander through this seeking method ignoring its handling of index
tiers, you'll end up eventually in the binary search of the index using our
key-conscious comparator:
http://hbase.apache.org/xref/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.html#482(Our
KeyComparator over in KeyValue implements RawComparator so the
RawComparator you see here is a base type -- we have different comparators
dependent on whether it user data or system table files... that is another
story).

Hope this gives you some insight on how we could skip to 'row1c' w/o going
through the content of 'row1' first.

You have a perf problem?  Maybe dump out content of a few hfiles that you
know you are having speed issues against.  It can be informative?

St.Ack
+
Stack 2013-05-16, 21:50
+
Varun Sharma 2013-05-17, 07:22
+
ramkrishna vasudevan 2013-05-17, 07:40
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB