Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Where is code in hbase that physically delete a record?


Copy link to this message
-
RE: Where is code in hbase that physically delete a record?
Ramkrishna.S.Vasudevan 2012-10-18, 04:20
Hi Yun

Hope Anoop's clear explanation will help you.
Just to add on, after you wrap the StoreScanner in your Custome Scanner Impl
you will invoke the next(List<KeyValue>) on the delegator(here the delegator
is the actual StoreScanner).
The delegator will give you the KV list that it has fetched from underlying
Scanners (Memstore and StoreFileScanner).
Now on the returned kv you can do a check say if the KV has a column C1 and
its value is 'a', just skip it so that this scanner does not send the kv to
the actual Scanner on the outside of the custom Scanner which the compaction
tries to use.

The Code may look lik this
Class CustomScanner implements InternalScanner{
StoreScanner delegate;
Public CustomScanner(){
Delegate = new SToreScanner();

Public boolean next(List<KeyValue>kv)
{
delegate.next(kv);
foreach(kv){
//Do necessary filtering here.
}

}
}

Regards
Ram

> -----Original Message-----
> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, October 18, 2012 9:02 AM
> To: [EMAIL PROTECTED]
> Subject: RE: Where is code in hbase that physically delete a record?
>
> Hi Yun,
>              We have preCompactScannerOpen() and preCompact() hooks..
> As we said, for compaction, a scanner for reading all corresponding
> HFiles ( all HFiles in major compaction) will be created and scan via
> that scanner.. ( calling next() methods).. The kernel will do this
> way..
> Now using these hooks you can create a wrapper over the actual
> scanner... In fact you can use preCompact() hook(I think that is fine
> for you).. By the time this is being called,  the actual scanner is
> made and will get that object passed to your hook... You can create  a
> custom scanner impl and wrap the actual scanner within that and return
> the new wrapper scanner from your post hook.. [Yes its return type is
> InternalScanner]  The actual scanner you can use as a delegator to do
> the actual scanning purpose... Now all the KVs ( which the underlying
> scanner passed) will flow via ur new wrapper scanner where you can
> avoid certain KVs based on your condition or logic
>
> Core                 WrapperScannerImpl                      Actual
> Scanner (created by core)
>         ->                 next(List<KeyValue>)     ->
> next(List<KeyValue>)
>                                                                     <-
> Do the real scan from HFiles
>                           See List KVs and remove
>                            those u dont want
>                                  <-
> Only the passed
> KVs come in final
> merged file
>
> Hope I make it clear for you :)
>
> Note : - preCompactScannerOpen() will be called before even creating
> the actual scanner while preCompact() after this scanner creation.. You
> can see the code in Store#compactStore()
>
> -Anoop-
> ________________________________________
> From: yun peng [[EMAIL PROTECTED]]
> Sent: Wednesday, October 17, 2012 9:04 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Where is code in hbase that physically delete a record?
>
> Hi, Ram and Anoop, Thanks for the nice reference on the java file,
> which I
> will check through.
>
> It is interesting to know about the recent feature on
> preCompactScannerOpen() hook. Ram, it would be nice if I can know how
> to
> specify conditions like c1 = 'a'.  I have also checked the example code
> in
> hbase 6496 link <https://issues.apache.org/jira/browse/HBASE-6496>.
> which
> show how to delete data before time as in a on-demand specification...
> Cheers,
> Yun
>
> On Wed, Oct 17, 2012 at 8:46 AM, Ramkrishna.S.Vasudevan <
> [EMAIL PROTECTED]> wrote:
>
> > Also to see the code how the delete happens pls refer to
> StoreScanner.java
> > and how the ScanQueryMatcher.match() works.
> >
> > That is where we decide if any kv has to be avoided due to already
> deleted
> > tombstone marker.
> >
> > Forgot to tell you about this.
> >
> > Regards
> > Ram
> >
> > > -----Original Message-----
> > > From: yun peng [mailto:[EMAIL PROTECTED]]