Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - EndPoint Coprocessor could be dealocked?


Copy link to this message
-
Re: EndPoint Coprocessor could be dealocked?
Michael Segel 2012-05-16, 19:07

 I think we need to look at the base problem that is trying to be solved.

I mean the discussion on the RPC mechanism. but the problem that the OP is trying to solve is how to use multiple indexes in a 'query'.

Note: I put ' ' around query because its a m/r job or a single thread where the user is trying to get a result set which is a significantly smaller subset, using more than 1 index.

So the idea is to do a quick get() against each index and the result would be a list of row keys. The next step is to get the intersection(s) quickly (which I proposed), and then you would just need to do a quick series of get()s  to pull back the list of rows.

If I understand the OP's problem, its not a co-processor type of problem.

Its one of where you submit a m/r job. Within your toolRunner, you would actually do the fetches against the indexes and then build the ultimate result set. then you just need a map job to take your result set as an input.

Drawback... if the list of rows is very, very long, you may run out of memory. So you need to resolve that...
(Which is why I was suggesting on using a temp table and then you can use the rows in the temp table as input in to your fetch...

While not something I would use for 'real time' its something where I can really shrink the number of rows you have to fetch for further processing.  So if your full table scan takes an hour, but we can do N get()s to get the rows in the Index, find the intersection I and then do I.size() get()s to fetch the data.  This should take much less time.
Again, I don't see this in a coprocessor based solution, however, the N get()s and intersection could be done at the start of the job, or could be part of a Map only job.

Kind of an interesting problem...  but if anyone has a large set of data and some time to play, you will end up solving a problem that you can' do in an RDBMS easily.

On May 16, 2012, at 1:17 PM, Andrew Purtell wrote:

>> On May 16, 2012, at 1:12 AM, fding hbase wrote:
>>> But sadly, HBase ipc doesn't allow coprocessor chaining mechanism...
>>> Someone mentioned on
>>> http://grokbase.com/t/hbase/user/116hrhhf8m/coprocessor-failure-question-and-examples
>>> :
>>>
>>> If a RegionObserver issues RPC to another table from any of the hooks that
>>> are called out of RPC handlers (for Gets, Puts, Deletes, etc.), you risk
>>> deadlock. Whatever activity you want to check should be in the same
>>> region as account data to avoid that.
>>> (Or HBase RPC needs to change.)
>>>
>>> So, that means, the deadlock is inevitable under current circumstance. The
>>> coprocessors are still limited.
>>>
>>> What I'm seeking is possible extensions of coprocessors or workaround for
>>> such situations that extra RPC is needed in the RPC handlers.
>
> This isn't a limitation, this is a design choice. Such extensions of
> coprocessors most likely won't happen. What a RegionObserver allows
> you to do is exactly this: Intercept and potentially modify lifecycle
> or user operations on that single region alone. If it helps, think of
> each region as its own independent database.
>
> If you need to take cross-region actions according to some user
> action, then you should be looking first at extending the client, not
> the server.
>
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet
> Hein (via Tom White)
>