Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Fwd: High CPU utilization in few Region servers during read


Copy link to this message
-
Re: Fwd: High CPU utilization in few Region servers during read
No this is different.

All your data is in the memstore still.

The memstore is organized as a skip list, nobody has ever tested that with 72gb. 256mb, 512mb, 1gb, sure... 72gb... no way.
Same with a 96gb of java heap. Not with Oracle or OpenJDK and an application specifically for such large heaps.
I would keep it under 30gb.
I think what you want is the following:
1. disable WAL writes (you don't care if you lose data)
2. lower your memstore size, so you'll see some flushes and eventually compactions.

3. Don't give the JVM more the 30g or so
4. Flush your memstores to disk. They'll end up in the block cache that way

Currently we can't fill the block cache without flushing to disk.

Maybe HBase is not the right solution. If you need a large ephemeral in-memory store, maybe look at memcache?
-- Lars

________________________________
 From: Vinay Kashyap <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Wednesday, October 23, 2013 5:57 PM
Subject: Fwd: High CPU utilization in few Region servers during read
 

>From the thread dump looks like so many threads are stuck at

org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1535)

org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1523)

java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(ConcurrentSkipListMap.java:647)

java.util.concurrent.ConcurrentSkipListMap.findNear(ConcurrentSkipListMap.java:1346)

Is this similar to HBASE-9428 issue.??

Waiting for some help regarding this.. :)
Thanks and regards
Vinay S Kashyap

---------- Forwarded message ----------
From: Vinay Kashyap <[EMAIL PROTECTED]>
Date: Tue, Oct 22, 2013 at 8:47 PM
Subject: High CPU utilization in few Region servers during read
To: [EMAIL PROTECTED]
Hi,

I am running HBase 0.94.6 (cdh-4.4.0) with 25 region servers.
I am testing a scenario to read and write only from/to RAM.

I have the following settings
Table precreated with 25 regions.
HFile size - 48 GB
MemStore size - 72 GB
Heap size - 96 GB

These settings are to avoid any flushes to the disk. Data need not be
persisted.

I am able to achieve a load throughput of 75K ops per region server.
While reading 23 region servers are serving requests with throughput of 55K
ops, but randomly 2 of the region servers always end up serving few 100 ops.

In these 2 region servers the CPU usage is very high and close to 100%
continuously bringing down the overall throughput. I did not observe any
long GC pauses in this time.

I also tried applying the patch for HBASE-9428 issue, but still faced the
same problem.
Thread dump for the affected region server is at
http://pastebin.com/JGx9gXnm

Any hints on how to solve this.?

Thanks and regards
Vinay S Kashyap