Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Hbase tuning for heavy write cluster


Copy link to this message
-
Re: Hbase tuning for heavy write cluster
 First, you want the RegionServer to use the available memory for caching, etc. Every byte of unused RAM is wasted.

I would make the heap slightly smaller than 32GB, so that the JVM can still use compressed OOPs.
So I'd set to 31GB.
Lastly, 800 writes/s still a bit low. How does the CPU usage look across the RegionServers?
If CPU is high, you might want to make the memstores *smaller* (it is expensive to read/write from/to a SkipList).
If you see bad IO, and many store files (as might be case following the discussion below) maybe you want to increase the memstores.

-- Lars

________________________________
 From: Rohit Dev <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, January 26, 2014 3:35 AM
Subject: Re: Hbase tuning for heavy write cluster
 

Hi Vladimir,

Here is my cluster status:

Cluster Size: 26
Server memory: 128GB
Total Writes per sec (data): 450 Mbps
Writes per sec (count) per server: avg ~800 writes/sec (some spikes
upto 3000 writes/sec)
Max Region Size: 16GB
Regions per server: ~140 (not sure if I would be able to merge some
empty regions while table is online)
We are running CDH 4.3

Recently I changed setttings to:
Java heap size for Region Server: 32GB
hbase.hregion.memstore.flush.size: 536870912
hbase.hstore.blockingStoreFiles: 30
hbase.hstore.compaction.max: 15
hbase.hregion.memstore.block.multiplier: 3
hbase.regionserver.maxlogs: 90 (it is too high for 512MB memstore flush size ?)

I'm seeing weird stuff, like one region has grown upto 34GB! and has
21 store files. MAX_FILESIZE for this table is only 16GB.
Could this be a problem ?

On Sat, Jan 25, 2014 at 9:49 PM, Vladimir Rodionov
<[EMAIL PROTECTED]> wrote:
> What is the load (ingestion) rate per server in your cluster?
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Rohit Dev [[EMAIL PROTECTED]]
> Sent: Saturday, January 25, 2014 6:09 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Hbase tuning for heavy write cluster
>
> Compaction queue is ~600 in one of the Region-Server, while it is less
> than 5 is others (total 26 nodes).
> Compaction queue started going up after I increased the settings[1].
> In general, one Major compaction takes about 18 Mins.
>
> In the same region-server I'm seeing these two log messages frequently:
>
> 2014-01-25 17:56:27,312 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs:
> logs=167, maxlogs=32; forcing flush of 1 regions(s):
> 3788648752d1c53c1ec80fad72d3e1cc
>
> 2014-01-25 17:57:48,733 INFO
> org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for
> 'IPC Server handler 53 on 60020' on region
> tsdb,\x008WR\xE2+\x90\x00\x00\x02Qu\xF1\x00\x00(\x00\x97A\x00\x008M(7\x00\x00Bl\xE85,1390623438462.e6692a1f23b84494015d111954bf00db.:
> memstore size 1.5 G is >= than blocking 1.5 G size
>
> Any suggestion what else I can do or is ok to ignore these messages ?
>
>
> [1]
> New settings are:
>  - hbase.hregion.memstore.flush.size - 536870912
>  - hbase.hstore.blockingStoreFiles - 30
>  - hbase.hstore.compaction.max - 15
>  - hbase.hregion.memstore.block.multiplier - 3
>
> On Sat, Jan 25, 2014 at 3:00 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> Yes, it is normal.
>>
>> On Jan 25, 2014, at 2:12 AM, Rohit Dev <[EMAIL PROTECTED]> wrote:
>>
>>> I changed these settings:
>>> - hbase.hregion.memstore.flush.size - 536870912
>>> - hbase.hstore.blockingStoreFiles - 30
>>> - hbase.hstore.compaction.max - 15
>>> - hbase.hregion.memstore.block.multiplier - 3
>>>
>>> Things seems to be getting better now, not seeing any of those
>>> annoying ' Blocking updates' messages. Except that, I'm seeing
>>> increase in 'Compaction Queue' size on some servers.
>>>
>>> I noticed memstores are getting flushed, but some with 'compaction
>>> requested=true'[1]. Is this normal ?
>>>
>>>
>>> [1]
>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB