Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Where is code in hbase that physically delete a record?


Copy link to this message
-
Re: Where is code in hbase that physically delete a record?
Yes the KVs coming out from your delegate Scanner will be in sorted form..
Also with all other logic applied like removing TTL expired data, handling
max versions etc..  Thanks for updating..

-Anoop-

On Sat, Oct 20, 2012 at 1:11 AM, PG <[EMAIL PROTECTED]> wrote:

> Hi, Anoop and Ram,
> As I have coded the idea,  the detailed instructions are very helpful. One
> minor thing to add is that coming out from scanner are the KeyValues which
> are already sorted by column qualifier and time stamps. though i did not
> find it mentioned in java doc, but i found it very useful feature to do
> filtering.
>
> Thanks.
> Yun
>
> On Oct 18, 2012, at 12:20 AM, "Ramkrishna.S.Vasudevan" <
> [EMAIL PROTECTED]> wrote:
>
> > Hi Yun
> >
> > Hope Anoop's clear explanation will help you.
> > Just to add on, after you wrap the StoreScanner in your Custome Scanner
> Impl
> > you will invoke the next(List<KeyValue>) on the delegator(here the
> delegator
> > is the actual StoreScanner).
> > The delegator will give you the KV list that it has fetched from
> underlying
> > Scanners (Memstore and StoreFileScanner).
> > Now on the returned kv you can do a check say if the KV has a column C1
> and
> > its value is 'a', just skip it so that this scanner does not send the kv
> to
> > the actual Scanner on the outside of the custom Scanner which the
> compaction
> > tries to use.
> >
> > The Code may look lik this
> > Class CustomScanner implements InternalScanner{
> > StoreScanner delegate;
> > Public CustomScanner(){
> > Delegate = new SToreScanner();
> >
> > Public boolean next(List<KeyValue>kv)
> > {
> >    delegate.next(kv);
> >    foreach(kv){
> >        //Do necessary filtering here.
> >    }
> >
> > }
> > }
> >
> > Regards
> > Ram
> >
> >> -----Original Message-----
> >> From: Anoop Sam John [mailto:[EMAIL PROTECTED]]
> >> Sent: Thursday, October 18, 2012 9:02 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: RE: Where is code in hbase that physically delete a record?
> >>
> >> Hi Yun,
> >>             We have preCompactScannerOpen() and preCompact() hooks..
> >> As we said, for compaction, a scanner for reading all corresponding
> >> HFiles ( all HFiles in major compaction) will be created and scan via
> >> that scanner.. ( calling next() methods).. The kernel will do this
> >> way..
> >> Now using these hooks you can create a wrapper over the actual
> >> scanner... In fact you can use preCompact() hook(I think that is fine
> >> for you).. By the time this is being called,  the actual scanner is
> >> made and will get that object passed to your hook... You can create  a
> >> custom scanner impl and wrap the actual scanner within that and return
> >> the new wrapper scanner from your post hook.. [Yes its return type is
> >> InternalScanner]  The actual scanner you can use as a delegator to do
> >> the actual scanning purpose... Now all the KVs ( which the underlying
> >> scanner passed) will flow via ur new wrapper scanner where you can
> >> avoid certain KVs based on your condition or logic
> >>
> >> Core                 WrapperScannerImpl                      Actual
> >> Scanner (created by core)
> >>        ->                 next(List<KeyValue>)     ->
> >> next(List<KeyValue>)
> >>                                                                    <-
> >> Do the real scan from HFiles
> >>                          See List KVs and remove
> >>                           those u dont want
> >>                                 <-
> >> Only the passed
> >> KVs come in final
> >> merged file
> >>
> >> Hope I make it clear for you :)
> >>
> >> Note : - preCompactScannerOpen() will be called before even creating
> >> the actual scanner while preCompact() after this scanner creation.. You
> >> can see the code in Store#compactStore()
> >>
> >> -Anoop-
> >> ________________________________________
> >> From: yun peng [[EMAIL PROTECTED]]
> >> Sent: Wednesday, October 17, 2012 9:04 PM
> >> To: [EMAIL PROTECTED]
> >> Subject: Re: Where is code in hbase that physically delete a record?