Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - RegionServers Crashing every hour in production env


Copy link to this message
-
Re: RegionServers Crashing every hour in production env
Pablo Musa 2013-03-12, 15:43
Guys,
thank you very much for the help.

Yesterday I spent 14 hours trying to tune the whole cluster.
The cluster is not ready yet needs a lot of tunning, but at least is
working.

My first big problem was namenode + datanode GC. They were not using
CMS and thus were taking "incremental" time to run. Ii started in 0.01
ms and
in 20 minutes was taking 150 secs.
After setting CMSGC this time is much smaller taking a maximum of 70 secs,
which is VERY HIGH, but for now does not stop HBase.

With this issue solved, it was clear that the RS was doing a long pause GC,
taking up to 220 secs. Zookeeper expired the RS and it shutdown.
I tried a lot of different flags configuration (MORE than 20), and could not
get small gcs. Eventually it would take more than 150 secs (zookeeper
timeout)
and shutdown.

Finally I tried a config that so far, 12 hours, is working with a maximum GC
time of 90 secs. Which of course is a terrible problem since HBase is a
database, but at least the cluster is stable while I can tune it a
little more.

In my opinion, my biggest problem is to have a few "monster" machines in the
cluster instead of a bunch of commodities machines. I don't know if
there are
a lot companies using this kind of machines inside a hadoop cluster, but
a fast search on google could not find a lot of tunes for big heap GCs.

I guess my next step will be search for big heap gc tuning.

Back to some questions ;)

 > You have ganglia or tsdb running?

I use zabbix for now, and no there is nothing going on when the big
pause happens.

 > When you see the big pause above, can you see anything going on on the
 > machine? (swap, iowait, concurrent fat mapreduce job?)
 > what are you doing during long GC happened? read or write? if
reading, what
 > the block cache size?

The cpu for the RS process goes to 100% and the logs "pause", until it
gets out.
Ex: [NewPar

IO and SWAP are normal. There is no MR running, just normal database
load, which is
very low. I am probably doing reads AND writes to the database with
default block
cache size.
One problem in this moment might be the big number of regions (1252)
since I am
using only one RS to be able to track the problem.

The links and ideas were very helpful. Thank you very much guys.

I will post my future researches as I find a solution ;)

If you have more ideas or info (links, flag suggestions, etc.), please
post it :)

Abs,
Pablo

On 03/10/2013 11:24 PM, Andrew Purtell wrote:
> Be careful with GC tuning, throwing changes at an application without
> analysis of what is going on with the heap is shooting in the dark. One
> particular good treatment of the subject is here:
> http://java.dzone.com/articles/how-tame-java-gc-pauses
>
> If you have made custom changes to blockcache or memstore configurations,
> back them out until you're sure everything else is ok.
>
> Watch carefully for swapping. Set the vm.swappiness sysctl to 0. Monitor
> for spikes in page scanning or any swap activity. Nothing brings on
> "Juliette" pauses better than a JVM partially swapped out. The Java GC
> starts collection by examining the oldest pages, and those are the first
> pages the OS swaps out...
>
>
>
> On Mon, Mar 11, 2013 at 10:13 AM, Azuryy Yu <[EMAIL PROTECTED]> wrote:
>
>> Hi Pablo,
>> It'a terrible for a long minor GC. I don't think there are swaping from
>> your vmstat log.
>> but I just suggest you
>> 1) add following JVM options:
>> -XX:+DisableExplicitGC -XX:+UseCompressedOops -XX:GCTimeRatio=19
>> -XX:SoftRefLRUPolicyMSPerMB=0 -XX:SurvivorRatio=2
>> -XX:MaxTenuringThreshold=3 -XX:+UseFastAccessorMethods
>>
>> 2) -Xmn is two small, your total Mem is 74GB, just make -Xmn2g
>> 3) what are you doing during long GC happened? read or write? if reading,
>> what the block cache size?
>>
>>
>>
>>
>> On Mon, Mar 11, 2013 at 6:41 AM, Stack <[EMAIL PROTECTED]> wrote:
>>
>>> You could increase your zookeeper session timeout to 5 minutes while you
>>> are figuring why these long pauses.
>>> http://hbase.apache.org/book.html#zookeeper.session.timeout