Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> skipKVsNewerThanReadpoint in StoreFileScanner and MemStoreScanner


Copy link to this message
-
Re: skipKVsNewerThanReadpoint in StoreFileScanner and MemStoreScanner
Created https://issues.apache.org/jira/browse/HBASE-9751.
On Fri, Oct 11, 2013 at 8:55 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> Taking a quick look... It is easy to reduce the ThreadLocal calls by 50%
> without any refactoring in the Memstore.
>
> Also thinking about this more, we already have metadata in HFile about the
> newest memstoreTS; I used that optimize storing the memstoreTS in HFiles
> away (HBASE-8166 and HBASE-8151).
>
> We could use this during scanning as well and avoid checking against the
> readpoint altogether.
> (might need some refactoring, though, have to checked closer).
>
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: lars hofhansl <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Cc:
> Sent: Friday, October 11, 2013 6:28 PM
> Subject: Re: skipKVsNewerThanReadpoint in StoreFileScanner and
> MemStoreScanner
>
> Interesting. Never came up in my profiling of raw scan performance.
> Of course my assumption was that the vast majority of all KVs are in
> HFiles and thus in the blockcache (not the memstore).
>
> ThreadLocals are indeed expensive, we ran into such problems as well (code
> unrelated to HBase, though).
>
>
> Mind filing a jira, Vladimir? If there's a perf issue, we should fix it.
>
>
> -- Lars
>
> ________________________________
>
> From: Vladimir Rodionov <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Friday, October 11, 2013 2:36 PM
> Subject: Re: skipKVsNewerThanReadpoint in StoreFileScanner and
> MemStoreScanner
>
>
> Yes, threadLocal.get() is expensive when you count every nanosecond. There
> is no need to call it every time in MemStoreScanner.next().
>
>
>
> On Fri, Oct 11, 2013 at 2:32 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
> > MultiVersionConsistencyControl.getThreadReadPoint() retrieves value from
> a
> > ThreadLocal.
> >
> > Can you specify which operation becomes expensive ?
> >
> > BTW I only found IsolationLevel.READ_UNCOMMITTED
> > in TestRowProcessorEndpoint. Looks like it is not normally used.
> >
> >
> > On Fri, Oct 11, 2013 at 2:20 PM, Vladimir Rodionov
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Update.
> > >
> > > This method is not called in MemStoreScanner, but direct call to :
> > >
> > > readPoint = MultiVersionConsistencyControl.getThreadReadPoint();
> > >
> > > on every next, seek etc
> > > which is basically the same.
> > >
> > > All these calls become very expensive when you try to achieve 10s of
> > > millions RPS in scan operations.
> > >
> > > Best regards,
> > > Vladimir Rodionov
> > > Principal Platform Engineer
> > > Carrier IQ, www.carrieriq.com
> > > e-mail: [EMAIL PROTECTED]
> > >
> > > ________________________________________
> > > From: Vladimir Rodionov
> > > Sent: Friday, October 11, 2013 2:18 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: skipKVsNewerThanReadpoint in StoreFileScanner and
> > MemStoreScanner
> > >
> > > From HRegion:
> > >
> > > synchronized(scannerReadPoints) {
> > >         if (isolationLevel == IsolationLevel.READ_UNCOMMITTED) {
> > >           // This scan can read even uncommitted transactions
> > >           this.readPt = Long.MAX_VALUE;
> > >
>    MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);
> > >         } else {
> > >           this.readPt > > > MultiVersionConsistencyControl.resetThreadReadPoint(mvcc);
> > >         }
> > >
> > > If isolation level is READ_UNCOMMITTED than this method should never be
> > > called in both StoreFileScanner and MemStoreScanner
> > > If isolation level is READ_COMMITTED than this method must be called
> only
> > > in MemStoreScanner, because HFiles are immutable no new KV's can be
> added
> > > to HFile after StoreFileScanner was created.
> > >
> > >
> > > Correct?
> > >
> > >
> > > Best regards,
> > > Vladimir Rodionov
> > > Principal Platform Engineer
> > > Carrier IQ, www.carrieriq.com
> > > e-mail: [EMAIL PROTECTED]
> > >
> > > Confidentiality Notice:  The information contained in this message,