Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> How to adjust hbase settings when too many store files?


Copy link to this message
-
Re: How to adjust hbase settings when too many store files?
What version of hbase were you using ?
Did you pre split the table before loading ?

Thanks

On Oct 28, 2012, at 8:33 PM, 苏铖 <[EMAIL PROTECTED]> wrote:

> Hello. I encounter a region server error when I try to put bulk data from a
> java client.
>
> The java client extracts data from a relational database and puts those data
> into hbase.
>
> When I try to extract data from a large table(say, 1 billion records), the
> error happens.
>
>
>
> The region server's log says:
>
>
>
>> 2012-10-28 00:00:02,169 WARN
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Region
> statistic_visit_detail,20120804|72495|8549
>
> 56,1351353594195.ad2592ee7a3610c60c47cf8be77496c8. has too many store files;
> delaying flush up to 90000ms
>
>> 2012-10-28 00:00:02,791 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
> because memory above low wa
>
> ter=347.1m
>
>> 2012-10-28 00:00:02,791 DEBUG
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Under global heap
> pressure: Region statistic_vis
>
> it_detail,20120804|72495|854956,1351353594195.ad2592ee7a3610c60c47cf8be77496
> c8. has too many store files, but is 141.5m vs best flus
>
> hable region's 46.8m. Choosing the bigger.
>
>> 2012-10-28 00:00:02,791 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush of region
> statistic_visit_detail,20120804|7
>
> 2495|854956,1351353594195.ad2592ee7a3610c60c47cf8be77496c8. due to global
> heap pressure
>
> ...
>
>
>
> And finally,
>
>
>
>> 2012-10-28 00:00:43,511 INFO org.apache.hadoop.hbase.regionserver.HRegion:
> compaction interrupted by user
>
>> java.io.InterruptedIOException: Aborting compaction of store cf1 in region
> statistic_visit_detail,20120804|72495|854956,135135359419
>
> 5.ad2592ee7a3610c60c47cf8be77496c8. because user requested stop.
>
>        at
> org.apache.hadoop.hbase.regionserver.Store.compactStore(Store.java:1275)
>
>        at
> org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:765)
>
>        at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1023)
>
>        at
> org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest.run(Compa
> ctionRequest.java:177)
>
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>        at java.lang.Thread.run(Thread.java:662)
>
>
>
> Then the region server shuts down.
>
>
>
> It seems that too many store files(due to too many records from
> relational-db) consumed too many memories, if I'm right.
>
> I'm new to hbase, what settings should I adjust? Or even increase region
> servers?
>
> I'm going to do some research by myself, and any advise will be appreciated.
>
> Best regards,
>
>
>
> Su
>
>
>