Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessors and batch processing


Copy link to this message
-
Re: Coprocessors and batch processing
On Wed, Aug 10, 2011 at 10:46 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

>
> I guess there could either be a {pre|post}Multi on RegionObserver (although
> HRegionServer.multi does a lot of munging).
> Or maybe a general {pre|post}Request with no arguments - in which case it
> would be at least possible to write code in the coprocessor
> to collect the puts/deletes/etc through the normal single
> prePut/preDelete/etc hooks and then batch-process them in postRequest().
>
>
This same question came up recently in an internal discussion as well.

Since multi() only exists at the HRegionServer level (there is no
HRegion.multi()), I don't think that this belongs in RegionObserver at all.
So this would be the main reason why there currently are no
preMulti()/postMulti() hooks.  There is something of a gap here between the
way the client sees things and the way the coprocessor sees things.  But I
really see this as more of an RPC-layer listener or filter.  After all,
"multi" is really just an RPC operation for efficiency.  It's not really a
core HBase operation.

As I see it there are a couple motivations behind the current limitation:

1) we generally only want a single set of coprocessor hooks to be involved
in a given operation
2) the coprocessor API should be as conceptually simple as possible, while
still reflecting what is happening in HBase
I think adding in some multi representation in the coprocessor API poses
challenges on each of these fronts.  Not necessarily unresolvable
challenges, but there are trade-offs involved.

Re (1): representing multi in cp hooks means that you now have layering in
the cp hooks handling a given operation.  Say all the actions in the multi
request are Puts, you go from having:

[prePut() ... postPut()] x N

to

preMulti() [prePut() ... postPut()] x N  postMulti()

For me, this implies some confusion about where I should handle the actions
in my coprocessor.  Should I put all handling in pre/postMulti() or do I
also need to implement pre/postPut()?

This gets at (2), for me it's easier to think of "multi" as just an
aggregation of operations, not as an operation in itself.  The actual
operation in the above example is a Put.  It's just that there are a lot of
them.
Back to the original situation you raise, I think it's a really bad idea to
immediately trigger RPC operations within a coprocessor, _especially_ in the
case of multi.  Say you are doing a secondary indexing transformation on the
Puts you receive.  You get a multi batch of 1000 puts.  You transform that
into a batch of 1000 secondary index puts, potentially going to every region
server in your cluster holding a region in the secondary indexing table.
You've just multiplied the RPC operations triggered by a single request and
exposed yourself to triggering a distributed deadlock, where the RPC handler
thread running in one RS is waiting for an RPC handler to become available
in another RS, which in turn has all handlers occupied waiting on other
servers.

I think the better approach to doing these kind of updates would be to have
the RegionObserver.pre/postPut() implementation queue them up and have them
batched and processed by a separate background thread so that you're not
tying up resources directly in the RPC handling path (and also making
clients wait on a synchronous response).

It may be that a higher level (RegionServer or RpcServer level)
observer-type interface would be useful.  But I think that adds some
complexity to understanding what coprocessors are and how they interact.

--gh