Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Hbase tuning for heavy write cluster


Copy link to this message
-
Re: Hbase tuning for heavy write cluster
Can you pastebin region server log corresponding to the 34GB region ?

Thanks

On Jan 26, 2014, at 3:35 AM, Rohit Dev <[EMAIL PROTECTED]> wrote:

> Hi Vladimir,
>
> Here is my cluster status:
>
> Cluster Size: 26
> Server memory: 128GB
> Total Writes per sec (data): 450 Mbps
> Writes per sec (count) per server: avg ~800 writes/sec (some spikes
> upto 3000 writes/sec)
> Max Region Size: 16GB
> Regions per server: ~140 (not sure if I would be able to merge some
> empty regions while table is online)
> We are running CDH 4.3
>
> Recently I changed setttings to:
> Java heap size for Region Server: 32GB
> hbase.hregion.memstore.flush.size: 536870912
> hbase.hstore.blockingStoreFiles: 30
> hbase.hstore.compaction.max: 15
> hbase.hregion.memstore.block.multiplier: 3
> hbase.regionserver.maxlogs: 90 (it is too high for 512MB memstore flush size ?)
>
> I'm seeing weird stuff, like one region has grown upto 34GB! and has
> 21 store files. MAX_FILESIZE for this table is only 16GB.
> Could this be a problem ?
>
>
> On Sat, Jan 25, 2014 at 9:49 PM, Vladimir Rodionov
> <[EMAIL PROTECTED]> wrote:
>> What is the load (ingestion) rate per server in your cluster?
>>
>> Best regards,
>> Vladimir Rodionov
>> Principal Platform Engineer
>> Carrier IQ, www.carrieriq.com
>> e-mail: [EMAIL PROTECTED]
>>
>> ________________________________________
>> From: Rohit Dev [[EMAIL PROTECTED]]
>> Sent: Saturday, January 25, 2014 6:09 PM
>> To: [EMAIL PROTECTED]
>> Subject: Re: Hbase tuning for heavy write cluster
>>
>> Compaction queue is ~600 in one of the Region-Server, while it is less
>> than 5 is others (total 26 nodes).
>> Compaction queue started going up after I increased the settings[1].
>> In general, one Major compaction takes about 18 Mins.
>>
>> In the same region-server I'm seeing these two log messages frequently:
>>
>> 2014-01-25 17:56:27,312 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Too many hlogs:
>> logs=167, maxlogs=32; forcing flush of 1 regions(s):
>> 3788648752d1c53c1ec80fad72d3e1cc
>>
>> 2014-01-25 17:57:48,733 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for
>> 'IPC Server handler 53 on 60020' on region
>> tsdb,\x008WR\xE2+\x90\x00\x00\x02Qu\xF1\x00\x00(\x00\x97A\x00\x008M(7\x00\x00Bl\xE85,1390623438462.e6692a1f23b84494015d111954bf00db.:
>> memstore size 1.5 G is >= than blocking 1.5 G size
>>
>> Any suggestion what else I can do or is ok to ignore these messages ?
>>
>>
>> [1]
>> New settings are:
>> - hbase.hregion.memstore.flush.size - 536870912
>> - hbase.hstore.blockingStoreFiles - 30
>> - hbase.hstore.compaction.max - 15
>> - hbase.hregion.memstore.block.multiplier - 3
>>
>> On Sat, Jan 25, 2014 at 3:00 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> Yes, it is normal.
>>>
>>> On Jan 25, 2014, at 2:12 AM, Rohit Dev <[EMAIL PROTECTED]> wrote:
>>>
>>>> I changed these settings:
>>>> - hbase.hregion.memstore.flush.size - 536870912
>>>> - hbase.hstore.blockingStoreFiles - 30
>>>> - hbase.hstore.compaction.max - 15
>>>> - hbase.hregion.memstore.block.multiplier - 3
>>>>
>>>> Things seems to be getting better now, not seeing any of those
>>>> annoying ' Blocking updates' messages. Except that, I'm seeing
>>>> increase in 'Compaction Queue' size on some servers.
>>>>
>>>> I noticed memstores are getting flushed, but some with 'compaction
>>>> requested=true'[1]. Is this normal ?
>>>>
>>>>
>>>> [1]
>>>> INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore
>>>> flush of ~512.0 M/536921056, currentsize=3.0 M/3194800 for region
>>>> tsdb,\x008ZR\xE1t\xC0\x00\x00\x02\x01\xB0\xF9\x00\x00(\x00\x0B]\x00\x008M((\x00\x00Bk\x9F\x0B,1390598160292.7fb65e5fd5c4cfe08121e85b7354bae9.
>>>> in 3422ms, sequenceid=18522872289, compaction requested=true
>>>>
>>>> On Fri, Jan 24, 2014 at 6:51 PM, Bryan Beaudreault
>>>> <[EMAIL PROTECTED]> wrote:
>>>>> Also, I think you can up the hbase.hstore.blockingStoreFiles quite a bit
>>>>> higher.  You could try something like 50.  It will reduce read performance

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB