Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase load problems


Copy link to this message
-
RE: HBase load problems
afaik,  your keys are already:
salted_prefix + timestamp

If cardinality of salted_prefix is not than big and it should not be (we use - 256 in our system) you will have already your rows grouped by
time in HFile blocks, I think you should have pretty pretty good locality of data in a block cache.
What is the cardinality of salted_prefix? And how many rows to  you ingest per day?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: Kireet [[EMAIL PROTECTED]]
Sent: Thursday, October 17, 2013 5:56 AM
To: [EMAIL PROTECTED]
Subject: Re: HBase load problems

Sorry for the confusion, I was trying to get at if it would be a large
operation or there would be a large disruption right after increasing
the max region size. It sounds like it wouldn't.

Getting back to the original question, if we do 90%+ of our reads/writes
to very recent data, would it make sense to keep that in a separate
table? It seems like that may keep things more optimized, the block
cache would be more efficient, compactions would run quicker with much
less data on the 'recent data' table and perhaps less often on the
'older data' table(s), etc.

On 10/17/13 12:52 AM, Stack wrote:
> On Wed, Oct 16, 2013 at 6:26 PM, Kireet <[EMAIL PROTECTED]> wrote:
>
>> is there a downside to going to larger regions?
>
>
>
> Generally we see pluses (See 2.5.6.2 Bigger Regions in
> http://hbase.apache.org/book/important_configurations.html for the latest
> scripture on the topic).  Downsides would be something like compactions run
> longer (coarsely, same overall work, just takes longer to complete a region)
>
>
>
>> It looks like merge is a larger operation, so changing the config setting
>> would simply cause existing regions to grow to the larger value right?
>> Would the way to confirm this be to check compaction rates and also maybe
>> the size of store files in hdfs?
>>
>>
> I'm not sure I understand the question above Kireet.  You doing merges?
>   Yes, you can just change the configs on the cluster and regions will just
> grow (will need to rolling restart the cluster to pick up the config.)
>
> Please ask again what you would confirm.
>
> Thanks,
> St.Ack
> P.S. Feedly-fan here.
>
Confidentiality Notice:  The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited.  If you have received this message in error, please immediately notify the sender and/or [EMAIL PROTECTED] and delete or destroy any copy of this message and its attachments.