Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # user - .META. region server DDOSed by too many clients


+
Varun Sharma 2012-12-05, 23:51
+
lars hofhansl 2012-12-06, 00:20
+
Varun Sharma 2012-12-06, 00:37
+
Varun Sharma 2012-12-06, 00:40
+
Anoop Sam John 2012-12-06, 04:25
+
ramkrishna vasudevan 2012-12-06, 06:04
+
Varun Sharma 2012-12-06, 10:32
+
ramkrishna vasudevan 2012-12-06, 10:59
+
Varun Sharma 2012-12-06, 11:29
+
ramkrishna vasudevan 2012-12-06, 11:52
Copy link to this message
-
Re: .META. region server DDOSed by too many clients
Varun Sharma 2012-12-06, 00:05
I am looking at a load of 4K QPS with 500 row updates in each RPC (so
almost 60K row updates) - I am using the batch interface so these should
get grouped into groups so that we only do 15 RPC(s) for each region server
- this is when the META is most needed. AFAIK, its cached at the client
side.

Also, is .META. specifically kept in memory - it seems the region server is
stuck in doing a seek - is this a sign of .META not being cached ? (PS: I
have one in memory column family - the block cache has 1G free in it).

On Wed, Dec 5, 2012 at 3:51 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I am running hbase 0.94.0 and I have a significant write load being put on
> a table with 98 regions on a 15 node cluster - also this write load comes
> from a very large number of clients (~ 1000). I am running with 10 priority
> IPC handlers and 200 IPC handlers. It seems the region server holding .META
> is DDOSed. All the 200 handlers are busy serving the .META. region and they
> are all locked onto on object. The Jstack is here for the regoin server
>
> "IPC Server handler 182 on 60020" daemon prio=10 tid=0x00007f329872c800
> nid=0x4401 waiting on condition [0x00007f328807f000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x0000000542d72e30> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
>         at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
>         at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
>         at
> java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:445)
>         at
> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:925)
>         at org.apache.hadoop.hbase.util.IdLock.getLockEntry(IdLock.java:71)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:290)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:455)
>         at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:493)
>         at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:242)
>         at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:167)
>         at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>         at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299)
>         at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244)
>         at
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521)
>         - locked <0x000000063b4965d0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
>         at
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402)
>         - locked <0x000000063b4965d0> (a
> org.apache.hadoop.hbase.regionserver.StoreScanner)
>         at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3354)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3310)
>         - locked <0x0000000523c211e0> (a
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl)