Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> .META. region server DDOSed by too many clients


Copy link to this message
-
Re: .META. region server DDOSed by too many clients
We only see this on the .META. region not otherwise...

On Wed, Dec 5, 2012 at 4:37 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> I see but is this pointing to the fact that we are heading to disk for
> scanning META - if yes, that would be pretty bad, no ? Currently I am
> trying to see if the freeze coincides with Block Cache being full (we have
> an inmemory column) - is the META table cached just like other tables ?
>
> Varun
>
>
> On Wed, Dec 5, 2012 at 4:20 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>
>> Looks like you're running into HBASE-5898.
>>
>>
>>
>> ----- Original Message -----
>> From: Varun Sharma <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Cc:
>> Sent: Wednesday, December 5, 2012 3:51 PM
>> Subject: .META. region server DDOSed by too many clients
>>
>> Hi,
>>
>> I am running hbase 0.94.0 and I have a significant write load being put on
>> a table with 98 regions on a 15 node cluster - also this write load comes
>> from a very large number of clients (~ 1000). I am running with 10
>> priority
>> IPC handlers and 200 IPC handlers. It seems the region server holding
>> .META
>> is DDOSed. All the 200 handlers are busy serving the .META. region and
>> they
>> are all locked onto on object. The Jstack is here for the regoin server
>>
>> "IPC Server handler 182 on 60020" daemon prio=10 tid=0x00007f329872c800
>> nid=0x4401 waiting on condition [0x00007f328807f000]
>>    java.lang.Thread.State: WAITING (parking)
>>         at sun.misc.Unsafe.park(Native Method)
>>         - parking to wait for  <0x0000000542d72e30> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>>         at
>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>>         at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)
>>         at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:871)
>>         at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1201)
>>         at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
>>         at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
>>         at
>>
>> java.util.concurrent.ConcurrentHashMap$Segment.put(ConcurrentHashMap.java:445)
>>         at
>>
>> java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:925)
>>         at
>> org.apache.hadoop.hbase.util.IdLock.getLockEntry(IdLock.java:71)
>>         at
>>
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:290)
>>         at
>>
>> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.seekToDataBlock(HFileBlockIndex.java:213)
>>         at
>>
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:455)
>>         at
>>
>> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:493)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:242)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:167)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:299)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:244)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:521)
>>         - locked <0x000000063b4965d0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:402)
>>         - locked <0x000000063b4965d0> (a
>> org.apache.hadoop.hbase.regionserver.StoreScanner)
>>         at
>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:127)