Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> .META. region server DDOSed by too many clients


Copy link to this message
-
Re: .META. region server DDOSed by too many clients
I am looking at a load of 4K QPS with 500 row updates in each RPC (so
almost 60K row updates) - I am using the batch interface so these should
get grouped into groups so that we only do 15 RPC(s) for each region server
- this is when the META is most needed. AFAIK, its cached at the client
side.

Also, is .META. specifically kept in memory - it seems the region server is
stuck in doing a seek - is this a sign of .META not being cached ? (PS: I
have one in memory column family - the block cache has 1G free in it).

On Wed, Dec 5, 2012 at 3:51 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am running hbase 0.94.0 and I have a significant write load being put on
> a table with 98 regions on a 15 node cluster - also this write load comes
> from a very large number of clients (~ 1000). I am running with 10 priority
> IPC handlers and 200 IPC handlers. It seems the region server holding .META
> is DDOSed. All the 200 handlers are busy serving the .META. region and they
> are all locked onto on object. The Jstack is here for the regoin server
>
> "IPC Server handler 182 on 60020" daemon prio=10 tid=0x00007f329872c800
> nid=0x4401 waiting on condition [0x00007f328807f000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x0000000542d72e30> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
>         at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
>         at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
>         at
> java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:445)
>         at
> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:925)
>         at org.apache.hadoop.hbase.util.IdLock.getLockEntry(IdLock.java:71)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:290)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:455)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:493)
>         at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:242)
>         at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:167)
>         at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>         at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299)
>         at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244)
>         at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521)
>         - locked <0x000000063b4965d0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
>         at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402)
>         - locked <0x000000063b4965d0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
>         at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3354)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3310)
>         - locked <0x0000000523c211e0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB