Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Sporadic memstore slowness for Read Heavy workloads


Copy link to this message
-
Re: Sporadic memstore slowness for Read Heavy workloads
I don't see bold line.
Is the following line what you referred to ?
org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(
MemStore.java:805)

BTW I assume you're using 0.94 HBase in production.

Cheers
On Sun, Jan 26, 2014 at 1:17 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:

> Here is the hotthread result. The line in bold is "SEEK_NEXT_COL" and it
> degenerates to a reseek+skiplist lookup when it does not have to because we
> already seeked to the row and getting all the columns is just a matter of
> iterator.next() calls.
>
>
>
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compareWithoutRow(KeyValue.java:2196)
>
> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:2086)
>
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1535)
>
> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1523)
>
>
> java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(ConcurrentSkipListMap.java:606)
>
>
> java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:685)
>
>
> java.util.concurrent.ConcurrentSkipListMap.findNear(ConcurrentSkipListMap.java:1345)
>
>
> java.util.concurrent.ConcurrentSkipListMap$SubMap.loNode(ConcurrentSkipListMap.java:2583)
>
>
> java.util.concurrent.ConcurrentSkipListMap$SubMap.access$300(ConcurrentSkipListMap.java:2486)
>
>
> java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapIter.<init>(ConcurrentSkipListMap.java:3022)
>
>
> java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapValueIterator.<init>(ConcurrentSkipListMap.java:3092)
>
>
> java.util.concurrent.ConcurrentSkipListMap$SubMap.valueIterator(ConcurrentSkipListMap.java:3002)
>
>
> java.util.concurrent.ConcurrentSkipListMap$Values.iterator(ConcurrentSkipListMap.java:2402)
>
>
> org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet.iterator(KeyValueSkipListSet.java:87)
>
>
> org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:805)
>
>
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>
>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:320)
>
>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:265)
>
>
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:545)
>
> *
>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:411)*
>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:143)
>
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3884)
>
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3956)
>
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3827)
>
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3808)
>
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3851)
>     org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4777)
>     org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750)
>
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152)
>
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700)
>
>
> On Sun, Jan 26, 2014 at 1:14 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > We are seeing some unfortunately low performance in the memstore - we
> have
> > researched some of the previous JIRA(s) and seen some inefficiencies in
> the
> > ConcurrentSkipListMap. The symptom is a RegionServer hitting 100 % cpu at
> > weird points in time - the bug is hard to reproduce and there isn't like
> a
> > huge # of extra reads going to that region server or any substantial
> > hotspot happening. The region server recovers the moment, we flush the
> > memstores or restart the region server. Our queries retrieve wide rows
> > which are upto 10-20 columns. A stack trace shows two things:

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB