Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - Improving Coprocessor postSplit/postOpen synchronization


Copy link to this message
-
RE: Improving Coprocessor postSplit/postOpen synchronization
Ramkrishna.S.Vasudevan 2012-08-29, 04:38
Hi

As per Andrew the postSplit needs to move up before even the transitioning
of the ZK nodes are done.  But in case of splits it is like if the META is
updated with the updated daughter info, the clients will tend to send in the
data to those updated regions.

So it is better we can add a new hook
postSplitBeforeDaughterOpeningDaughterRegions() (the name may be not very
correct, we can change it) just after the PONR step is completed.  This will
help the code in the CP to know what action to take for the region the CP
hooks are handling.  So I deprecated the postSplit() hook when I gave a
patch I think we can bring it back considering Andy's point and the current
usecase in discussion.

Regards
Ram

> -----Original Message-----
> From: Kevin Shin [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, August 29, 2012 2:30 AM
> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: Re: Improving Coprocessor postSplit/postOpen synchronization
>
> Hello again everyone,
>
> Thanks for responding! I really appreciate all of the advice that's
> been
> given so far.  :)
>
> Just to clarify Andrew do you have a prototype patch up that could
> potentially be worked on to either move postSplit() or add new hooks
> into
> the framework/are planning on submitting it sometime in the near
> future?
>
> I'd also love to get any feedback from the community about where to add
> the
> hook(s) but my thought was that we should have different levels of
> hooks
> within a split as Ramkrishna suggested. Perhaps two preSplits to
> accomodate
> for grabbing as well as a postSplit and a completeSplit? Giving a
> better
> abstraction would definitely help developers figure out how to deal
> with
> asynchronous calls to split, Put, and Delete. Thanks as always!
>
> Best,
> Kevin
>
> On Tue, Aug 28, 2012 at 11:12 AM, lars hofhansl <[EMAIL PROTECTED]>
> wrote:
>
> > That approach sounds good to me.
> >
> >
> >
> > ----- Original Message -----
> > From: Andrew Purtell <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Cc:
> > Sent: Tuesday, August 28, 2012 3:05 AM
> > Subject: Re: Improving Coprocessor postSplit/postOpen synchronization
> >
> > Never mind, I went to look at the code. Should have done that first.
> >
> > Looking at 0.94 sources, in SplitTransaction, first we notify the
> master
> > that the split has happened, and wait for the master to process it
> (which
> > opens daughters), and then call up to the CP with the daughter
> regions as
> > arguments.
> >
> > I seem to remember that in my prototype patch for the CP framework,
> > postSplit notification let the CP know the split took place and allow
> it to
> > take actions before the master opened the daughters. In any event
> that's
> > not the code now, so it seems what you need here is for us to move
> the
> > postSplit upcall up prior to master notification or add another hook
> at
> > that location.
> >
> > On Tue, Aug 28, 2012 at 12:53 PM, Andrew Purtell <[EMAIL PROTECTED]
> > >wrote:
> >
> > > (from postSplit)
> > >
> > >
> > > On Tue, Aug 28, 2012 at 12:53 PM, Andrew Purtell
> <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> What about writing a marker (a file) into the region at split
> (from
> > >> preSplit) which is then existence checked and read at open
> (postOpen)?
> > This
> > >> file would contain whatever indexing metadata is required.
> > >>
> > >> Also, splits are nearly instant because the daughters are created
> with
> > >> reference files to the parent, until a later compaction brings the
> data
> > >> from the parent over. Can you do the same with your indexes?
> Reason I
> > ask
> > >> is this notion of "ignoring" new data until indexes are available
> seems
> > >> undesirable.
> > >>
> > >>
> > >> On Mon, Aug 27, 2012 at 11:29 PM, Kevin Shin <
> > >> [EMAIL PROTECTED]> wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> A colleague and I were working with HBase coprocessors for
> secondary
> > >>> indexes and ran into an interesting problem regarding splits