Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Re: Hbase tuning for heavy write cluster


Copy link to this message
-
Re: Hbase tuning for heavy write cluster
Ted Yu 2014-01-26, 12:31
Can you pastebin region server log corresponding to the 34GB region ?

Thanks

On Jan 26, 2014, at 3:35 AM, Rohit Dev <[EMAIL PROTECTED]> wrote:

> Hi Vladimir,
>
> Here is my cluster status:
>
> Cluster Size: 26
> Server memory: 128GB
> Total Writes per sec (data): 450 Mbps
> Writes per sec (count) per server: avg ~800 writes/sec (some spikes
> upto 3000 writes/sec)
> Max Region Size: 16GB
> Regions per server: ~140 (not sure if I would be able to merge some
> empty regions while table is online)
> We are running CDH 4.3
>
> Recently I changed setttings to:
> Java heap size for Region Server: 32GB
> hbase.hregion.memstore.flush.size: 536870912
> hbase.hstore.blockingStoreFiles: 30
> hbase.hstore.compaction.max: 15
> hbase.hregion.memstore.block.multiplier: 3
> hbase.regionserver.maxlogs: 90 (it is too high for 512MB memstore flush size ?)
>
> I'm seeing weird stuff, like one region has grown upto 34GB! and has
> 21 store files. MAX_FILESIZE for this table is only 16GB.
> Could this be a problem ?
>
>
> On Sat, Jan 25, 2014 at 9:49 PM, Vladimir Rodionov
> <[EMAIL PROTECTED]> wrote:
>> What is the load (ingestion) rate per server in your cluster?
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: [EMAIL PROTECTED]
>>
>> ________________________________________
>> From: Rohit Dev [[EMAIL PROTECTED]]
>> Sent: Saturday, January 25, 2014 6:09 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Hbase tuning for heavy write cluster
>>
>> Compaction queue is ~600 in one of the Region-Server, while it is less
>> than 5 is others (total 26 nodes).
>> Compaction queue started going up after I increased the settings[1].
>> In general, one Major compaction takes about 18 Mins.
>>
>> In the same region-server I'm seeing these two log messages frequently:
>>
>> 2014-01-25 17:56:27,312 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs:
>> logs=167, maxlogs=32; forcing flush of 1 regions(s):
>> 3788648752d1c53c1ec80fad72d3e1cc
>>
>> 2014-01-25 17:57:48,733 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for
>> 'IPC Server handler 53 on 60020' on region
>> tsdb,\x008WR\xE2+\x90\x00\x00\x02Qu\xF1\x00\x00(\x00\x97A\x00\x008M(7\x00\x00Bl\xE85,1390623438462.e6692a1f23b84494015d111954bf00db.:
>> memstore size 1.5 G is >= than blocking 1.5 G size
>>
>> Any suggestion what else I can do or is ok to ignore these messages ?
>>
>>
>> [1]
>> New settings are:
>> - hbase.hregion.memstore.flush.size - 536870912
>> - hbase.hstore.blockingStoreFiles - 30
>> - hbase.hstore.compaction.max - 15
>> - hbase.hregion.memstore.block.multiplier - 3
>>
>> On Sat, Jan 25, 2014 at 3:00 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> Yes, it is normal.
>>>
>>> On Jan 25, 2014, at 2:12 AM, Rohit Dev <[EMAIL PROTECTED]> wrote:
>>>
>>>> I changed these settings:
>>>> - hbase.hregion.memstore.flush.size - 536870912
>>>> - hbase.hstore.blockingStoreFiles - 30
>>>> - hbase.hstore.compaction.max - 15
>>>> - hbase.hregion.memstore.block.multiplier - 3
>>>>
>>>> Things seems to be getting better now, not seeing any of those
>>>> annoying ' Blocking updates' messages. Except that, I'm seeing
>>>> increase in 'Compaction Queue' size on some servers.
>>>>
>>>> I noticed memstores are getting flushed, but some with 'compaction
>>>> requested=true'[1]. Is this normal ?
>>>>
>>>>
>>>> [1]
>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore
>>>> flush of ~512.0 M/536921056, currentsize=3.0 M/3194800 for region
>>>> tsdb,\x008ZR\xE1t\xC0\x00\x00\x02\x01\xB0\xF9\x00\x00(\x00\x0B]\x00\x008M((\x00\x00Bk\x9F\x0B,1390598160292.7fb65e5fd5c4cfe08121e85b7354bae9.
>>>> in 3422ms, sequenceid=18522872289, compaction requested=true
>>>>
>>>> On Fri, Jan 24, 2014 at 6:51 PM, Bryan Beaudreault
>>>> <[EMAIL PROTECTED]> wrote:
>>>>> Also, I think you can up the hbase.hstore.blockingStoreFiles quite a bit
>>>>> higher.  You could try something like 50.  It will reduce read performance