Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Re: Hbase tuning for heavy write cluster


Copy link to this message
-
Re: Hbase tuning for heavy write cluster
 First, you want the RegionServer to use the available memory for caching, etc. Every byte of unused RAM is wasted.

I would make the heap slightly smaller than 32GB, so that the JVM can still use compressed OOPs.
So I'd set to 31GB.
Lastly, 800 writes/s still a bit low. How does the CPU usage look across the RegionServers?
If CPU is high, you might want to make the memstores *smaller* (it is expensive to read/write from/to a SkipList).
If you see bad IO, and many store files (as might be case following the discussion below) maybe you want to increase the memstores.

-- Lars

________________________________
 From: Rohit Dev <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, January 26, 2014 3:35 AM
Subject: Re: Hbase tuning for heavy write cluster
 

Hi Vladimir,

Here is my cluster status:

Cluster Size: 26
Server memory: 128GB
Total Writes per sec (data): 450 Mbps
Writes per sec (count) per server: avg ~800 writes/sec (some spikes
upto 3000 writes/sec)
Max Region Size: 16GB
Regions per server: ~140 (not sure if I would be able to merge some
empty regions while table is online)
We are running CDH 4.3

Recently I changed setttings to:
Java heap size for Region Server: 32GB
hbase.hregion.memstore.flush.size: 536870912
hbase.hstore.blockingStoreFiles: 30
hbase.hstore.compaction.max: 15
hbase.hregion.memstore.block.multiplier: 3
hbase.regionserver.maxlogs: 90 (it is too high for 512MB memstore flush size ?)

I'm seeing weird stuff, like one region has grown upto 34GB! and has
21 store files. MAX_FILESIZE for this table is only 16GB.
Could this be a problem ?

On Sat, Jan 25, 2014 at 9:49 PM, Vladimir Rodionov
<[EMAIL PROTECTED]> wrote:
> What is the load (ingestion) rate per server in your cluster?
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Rohit Dev [[EMAIL PROTECTED]]
> Sent: Saturday, January 25, 2014 6:09 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Hbase tuning for heavy write cluster
>
> Compaction queue is ~600 in one of the Region-Server, while it is less
> than 5 is others (total 26 nodes).
> Compaction queue started going up after I increased the settings[1].
> In general, one Major compaction takes about 18 Mins.
>
> In the same region-server I'm seeing these two log messages frequently:
>
> 2014-01-25 17:56:27,312 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs:
> logs=167, maxlogs=32; forcing flush of 1 regions(s):
> 3788648752d1c53c1ec80fad72d3e1cc
>
> 2014-01-25 17:57:48,733 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for
> 'IPC Server handler 53 on 60020' on region
> tsdb,\x008WR\xE2+\x90\x00\x00\x02Qu\xF1\x00\x00(\x00\x97A\x00\x008M(7\x00\x00Bl\xE85,1390623438462.e6692a1f23b84494015d111954bf00db.:
> memstore size 1.5 G is >= than blocking 1.5 G size
>
> Any suggestion what else I can do or is ok to ignore these messages ?
>
>
> [1]
> New settings are:
>  - hbase.hregion.memstore.flush.size - 536870912
>  - hbase.hstore.blockingStoreFiles - 30
>  - hbase.hstore.compaction.max - 15
>  - hbase.hregion.memstore.block.multiplier - 3
>
> On Sat, Jan 25, 2014 at 3:00 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> Yes, it is normal.
>>
>> On Jan 25, 2014, at 2:12 AM, Rohit Dev <[EMAIL PROTECTED]> wrote:
>>
>>> I changed these settings:
>>> - hbase.hregion.memstore.flush.size - 536870912
>>> - hbase.hstore.blockingStoreFiles - 30
>>> - hbase.hstore.compaction.max - 15
>>> - hbase.hregion.memstore.block.multiplier - 3
>>>
>>> Things seems to be getting better now, not seeing any of those
>>> annoying ' Blocking updates' messages. Except that, I'm seeing
>>> increase in 'Compaction Queue' size on some servers.
>>>
>>> I noticed memstores are getting flushed, but some with 'compaction
>>> requested=true'[1]. Is this normal ?
>>>
>>>
>>> [1]
>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore