Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> KeyValue.getLength() question


Copy link to this message
-
Re: KeyValue.getLength() question
Are you on the client or the server?
In the server the KeyValue objects are created in HFileReaderV2.ScannerV2.getKeyValue(). There you will see that a KeyValue object is really just a "pointer" into a larger byte[] loaded from an HFile.

On the client the KeyValue is typically deserialized from an RPC; in that case the backing array only holds one KeyValue (and the buffer size and the KeyValue length should match).
Does that make sense? I know this can be a bit confusing.
-- Lars

----- Original Message -----
From: Kim Chew <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
Cc:
Sent: Wednesday, September 25, 2013 5:40 PM
Subject: Re: KeyValue.getLength() question

On Wed, Sep 25, 2013 at 7:52 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> myKV.getLength() is alway <= myKV.getBuffer().length.
>
> The buffer here is typically an HFile block.
    Lars, I don't quite understand this, could you please elaborate a bit
more? Also if the KV's buffer size is bigger than the one returned by
"readLength()", what would be those extra bytes in the buffer?

    It seems to me that the Scanner and InternalScanner packs different
numbers of extra bytes to the buffer, I tired to pinpoint the scanner codes
to where the KV objects is created but without too much luck. Could you
show me where it is done?

Thanks a lot.

Kim
> We use that buffer and pass it up the chain without making any further
> copy of the KV.
>

>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Kim Chew <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Wednesday, September 25, 2013 12:06 AM
> Subject: KeyValue.getLength() question
>
> Hello,
>
> I have a "strange" situation that I can't wrap my head around it. Say, for
> example, I have an KeyValue instance, shouldn't
>
>     myKV.getLength() == myKV.getBuffer().length ?
>
> Given that, "getLength()" returns "Length of bytes this KeyValue occupies
> in getBuffer()<
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/KeyValue.html#getBuffer%28%29
> >
> ."
>
>
> In my case the value returned by "myKV.getBuffer().length" is greater than
> "myKV.getLength()". What possibly went wrong?
>
> TIA
>
> Kim.
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB