Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Problems with hbase.hregion.max.filesize


Copy link to this message
-
Re: Problems with hbase.hregion.max.filesize
Did you observe anything interesting with such a large Java heap?
You said you have 3G for the memstore, most of the rest is for the block cache I assume.
Any long GC pauses, or other strange behavior?

Thanks.

-- Lars

________________________________
 From: Timo Schaepe <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Saturday, December 14, 2013 5:27 AM
Subject: Re: Problems with hbase.hregion.max.filesize
 

Sorry, forgot our Hardwareconfiguration…

1 NameNode/SecondaryNameNode/HBase master
31 Datanodes/Regionserver

All of them with
2x XEON E5-2640 2.5 GHz
128 GB RAM
/dev/sda 90 GB
/dev/sdb 1.1 TB
/dev/sdc 1.1 TB

where sda are SSDs disks for system and sdb and sdc are disks for HDFS/HBase

Heapsize for Regionserver: 80 GB

bye,

    Timo

Am 14.12.2013 um 14:21 schrieb Timo Schaepe <[EMAIL PROTECTED]>:

> Hey,
>
> @JM: Thanks for the hint with hbase.regionserver.fileSplitTimeout. At the moment (the import is actually working) and after I splittet the specific regions manually, we do not have growing regions anymore.
>
> hbase hbck says, all things are going fine.
> 0 inconsistencies detected.
> Status: OK
>
> @Ted Yu: Sure, have a look here: http://pastebin.com/2ANFVZEU
> The relevant tablename ist data_1091.
>
> Thanks for your time.
>
>     Timo
>
> Am 13.12.2013 um 20:18 schrieb Ted Yu <[EMAIL PROTECTED]>:
>
>> Timo:
>> Can you pastebin regionserver log around 2013-12-12 13:54:20 so that we can
>> see what happened ?
>>
>> Thanks
>>
>>
>> On Fri, Dec 13, 2013 at 11:02 AM, Jean-Marc Spaggiari <
>> [EMAIL PROTECTED]> wrote:
>>
>>> Try to increase hbase.regionserver.fileSplitTimeout but put it back to its
>>> default value after.
>>>
>>> Default value is 30 seconds. I think it's not normal for a split to take
>>> more than that.
>>>
>>> What is your hardware configuration?
>>>
>>> Have you run hbck to see if everything is correct?
>>>
>>> JM
>>>
>>>
>>> 2013/12/13 Timo Schaepe <[EMAIL PROTECTED]>
>>>
>>>> Hello again,
>>>>
>>>> digging in the logs of the specific regionserver shows me that:
>>>>
>>>> 2013-12-12 13:54:20,194 INFO
>>>> org.apache.hadoop.hbase.regionserver.SplitRequest: Running
>>> rollback/cleanup
>>>> of failed split of
>>>>
>>> data,OR\x83\xCF\x02\x82\xAE\xF3U,1386851456415.d04cadb1b2252dafc476c138e9651ca7.;
>>>> Took too long to split the files and create the references, aborting
>>> split
>>>>
>>>> This message appears two time, so it seems, that HBase tried to split the
>>>> region but it failed. I don't know why. How is the behaviour of HBase,
>>> if a
>>>> region split fails? Are there more tries to split this region again? I
>>>> didn't find any new tries in the log. Now I split the big regions
>>> manually
>>>> and this works. And also it seems, that HBase split the new regions again
>>>> to crunch they down to the given limit.
>>>>
>>>> But also it is a mystery for me, why the split size in Hannibal shows me
>>>> 10 GB and in base-site.xml I put 2 GB…
>>>>
>>>> Thanks,
>>>>
>>>>       Timo
>>>>
>>>>
>>>> Am 13.12.2013 um 10:22 schrieb Timo Schaepe <[EMAIL PROTECTED]>:
>>>>
>>>>> Hello,
>>>>>
>>>>> during the loading of data in our cluster I noticed some strange
>>>> behavior of some regions, that I don't understand.
>>>>>
>>>>> Scenario:
>>>>> We convert data from a mysql database to HBase. The data is inserted
>>>> with a put to the specific HBase table. The row key is a timestamp. I
>>> know
>>>> the problem with timestamp keys, but in our requirement it works quiet
>>>> well. The problem is now, that there are some regions, which are growing
>>>> and growing.
>>>>>
>>>>> For example the table on the picture [1]. First, all data was
>>>> distributed over regions and node. And now, the data is written into only
>>>> one region, which is growing and I can see no splitting at all. Actually
>>>> the size of the big region is nearly 60 GB.
>>>>>
>>>>> HBase version is 0.94.11. I cannot understand, why the splitting is not
>>>> happening. In hbase-site.xml I limit the hbase.hregion.max.filesize to 2
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB