Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1


+
Vincent Barat 2012-11-20, 16:21
+
Stack 2012-11-21, 05:05
+
Vincent Barat 2012-11-21, 08:23
+
Alok Singh 2012-11-21, 04:53
+
Vincent Barat 2012-11-21, 09:02
+
Vincent Barat 2012-11-21, 09:04
Copy link to this message
-
Re: X3 slow down after moving from HBase 0.90.3 to HBase 0.92.1
On Wed, Nov 21, 2012 at 1:04 AM, Vincent Barat <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I've checked my 30 RPC handlers, they are all in a WAITING state:
>
> Thread 89 (PRI IPC Server handler 6 on 60020):
>   State: WAITING
>   Blocked count: 238
>   Waited count: 617
>   Waiting on
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131f139b
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1299)
>

So Vincent, the servers are quiet?   Which would match your low CPU
observation.  Clients are unable to send them load for some reason?
How many disks.  What is your block cache hit number (see regionserver
log -- it gets printed every so often .... or in the below I see 99%
so your numbers should be good coming out of the regionserver).

>
>
> Here is some extract for one of our RS (this is similar to all of them):
>
> requestsPerSecond=593, numberOfOnlineRegions=584, numberOfStores=1147,
> numberOfStorefiles=1980, storefileIndexSizeMB=15, rootIndexSizeKB=16219,
> totalStaticIndexSizeKB=246127, totalStaticBloomSizeKB=12936,
> memstoreSizeMB=1421, readRequestsCount=633241097,
> writeRequestsCount=9375846, compactionQueueSize=0, flushQueueSize=0,
> usedHeapMB=3042, maxHeapMB=4591, blockCacheSizeMB=890.19,
> blockCacheFreeMB=257.65, blockCacheCount=14048,
> blockCacheHitCount=5854936149, blockCacheMissCount=14761288,
> blockCacheEvictedCount=4870523, blockCacheHitRatio=99%,
> blockCacheHitCachingRatio=99%, hdfsBlocksLocalityIndex=29
>

600 regions is a lot per server.  You should put it on your TODO list
to have less per server -- bigger regions which you can do now you are
on 0.92.

If you major compact -- do it when site is less heavily loaded -- does
our performance go up.

Are all query types slow or just certain types?

St.Ack
+
Vincent Barat 2012-11-21, 18:35
+
Vincent Barat 2012-11-21, 08:18