Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Sporadic memstore slowness for Read Heavy workloads


Copy link to this message
-
Re: Sporadic memstore slowness for Read Heavy workloads
On a second look, this is the bold line:

org.apache.hadoop.hbase.regionserver.StoreScanner.
next(StoreScanner.java:411)*
On Sun, Jan 26, 2014 at 1:43 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> I don't see bold line.
> Is the following line what you referred to ?
> org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(
> MemStore.java:805)
>
> BTW I assume you're using 0.94 HBase in production.
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 1:17 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:
>
>> Here is the hotthread result. The line in bold is "SEEK_NEXT_COL" and it
>> degenerates to a reseek+skiplist lookup when it does not have to because
>> we
>> already seeked to the row and getting all the columns is just a matter of
>> iterator.next() calls.
>>
>>
>>
>> org.apache.hadoop.hbase.KeyValue$KeyComparator.compareWithoutRow(KeyValue.java:2196)
>>
>> org.apache.hadoop.hbase.KeyValue$KeyComparator.compare(KeyValue.java:2086)
>>
>> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1535)
>>
>> org.apache.hadoop.hbase.KeyValue$KVComparator.compare(KeyValue.java:1523)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(ConcurrentSkipListMap.java:606)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap.findPredecessor(ConcurrentSkipListMap.java:685)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap.findNear(ConcurrentSkipListMap.java:1345)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap$SubMap.loNode(ConcurrentSkipListMap.java:2583)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap$SubMap.access$300(ConcurrentSkipListMap.java:2486)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapIter.<init>(ConcurrentSkipListMap.java:3022)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap$SubMap$SubMapValueIterator.<init>(ConcurrentSkipListMap.java:3092)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap$SubMap.valueIterator(ConcurrentSkipListMap.java:3002)
>>
>>
>> java.util.concurrent.ConcurrentSkipListMap$Values.iterator(ConcurrentSkipListMap.java:2402)
>>
>>
>> org.apache.hadoop.hbase.regionserver.KeyValueSkipListSet.iterator(KeyValueSkipListSet.java:87)
>>
>>
>> org.apache.hadoop.hbase.regionserver.MemStore$MemStoreScanner.reseek(MemStore.java:805)
>>
>>
>> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:54)
>>
>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:320)
>>
>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:265)
>>
>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:545)
>>
>> *
>>
>> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:411)*
>>
>> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:143)
>>
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3884)
>>
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3956)
>>
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3827)
>>
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3808)
>>
>>
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3851)
>>     org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4777)
>>     org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4750)
>>
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2152)
>>
>>
>> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3700)
>>
>>
>>
>> On Sun, Jan 26, 2014 at 1:14 PM, Varun Sharma <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hi,
>> >
>> > We are seeing some unfortunately low performance in the memstore - we
>> have
>> > researched some of the previous JIRA(s) and seen some inefficiencies in
>> the
>> > ConcurrentSkipListMap. The symptom is a RegionServer hitting 100 % cpu

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB