Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Design review: Secondary index support through coprocessors


Copy link to this message
-
RE: Design review: Secondary index support through coprocess
Thanks, Vladimir. So a RPC call RS1 --> RS2 takes two handlers, one from
RS1 and one from RS2? If that is true, then I understand that it is a
typical Dining philosophers problem.

Maybe a random yielding mechanism can solve this problem.
Best regards,
Wei

---------------------------------
Wei Tan, PhD
Research Staff Member
IBM T. J. Watson Research Center
http://researcher.ibm.com/person/us-wtan

From:   Vladimir Rodionov <[EMAIL PROTECTED]>
To:     "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>,
Date:   01/22/2014 12:09 PM
Subject:        RE: Design review: Secondary index support through
coprocess

Deadlocks are possible because  cross region RPCs create cyclic
dependencies in HBase cluster.

RS1-> RS2->RS3->RS1, where -> is PRC call

now imagine that last call from RS3 to RS1 is blocked because there no
more available handler threads to process it.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: Wei Tan [[EMAIL PROTECTED]]
Sent: Wednesday, January 22, 2014 7:51 AM
To: [EMAIL PROTECTED]
Subject: RE: Design review: Secondary index support through coprocess

Why cross-RS RPC is going to cause deadlocks? It is a matter of logic
incorrectness, or resource outage? Say, if we set the #handler to be
large, logically deadlock still occurs?
Best regards,
Wei
From:   Vladimir Rodionov <[EMAIL PROTECTED]>
To:     "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>,
Date:   01/20/2014 03:00 PM
Subject:        RE: Design review: Secondary index support through
coprocess

>>Yes, the coprocessors potentially cross RS boundaries.

The open path to the disaster. Inter region RPCs in coprocessors may
result in periodic cluster - wide deadlocks
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: James Taylor [[EMAIL PROTECTED]]
Sent: Monday, January 20, 2014 11:39 AM
To: [EMAIL PROTECTED]
Subject: Re: Design review: Secondary index support through coprocess

Yes, the coprocessors potentially cross RS boundaries. No, the index is
not
co-located with the main table. Take a look at the link I sent as that
should be able to answer a lot of questions.

Thanks,
James
On Mon, Jan 20, 2014 at 11:03 AM, Michael Segel
<[EMAIL PROTECTED]>wrote:

> James,
>
> Ok…
>
> Its been a while since we talked about this…
>
> While the index is in a separate table, is that table being split and
> collocated with the main table?
>
> If you’re using the coprocessor to maintain the index, that would imply
> you’re crossing RS boundaries if your index is truly orthogonal.
>
> Is this what you’re doing?
>
> On Jan 20, 2014, at 11:32 AM, James Taylor <[EMAIL PROTECTED]>
wrote:
>
> > Mike,
> > Yes, you're mistaken:
> > - secondary indexes in Phoenix are orthogonal to the base table.
They're
> in
> > a separate table (
> > http://phoenix.incubator.apache.org/secondary_indexing.html).
> > - Phoenix has joins. They're in our master branch with a release
> scheduled
> > for next month
> > - numeric strings? Not a use case for indexing numeric data? Have you
> ever
> > seen a number used as an ID?
> > Thanks,
> > James
> >
> >
> > On Mon, Jan 20, 2014 at 8:50 AM, Michael Segel <
> [EMAIL PROTECTED]>wrote:
> >
> >> Indexes tend to be orthogonal to the base table, not to mention if
> you’re
> >> using an inverted table for an index, your index table would be much
> >> thinner than your base table.
> >>
> >> Having said that, the solution proposed by Yu, Taylor and others only
> >> works if you want to use the index to help on server side filtering
and
> >> misses the boat on the larger and broader picture of improving query
> >> optimization and joins.
> >>
> >> HINT: Unless I am mistaken… until you treat the index as orthogonal
to
> the
> >> base table, you will always lag performance of traditional MPP DWs
like
separate
first
take
sort
what
Take
with
has
if
Once
to
Phoenix
"how
that's a
in
away
is
behind
Phoenix
indexing
SI
several
LEXIN)
https://issues.apache.org/jira/secure/attachment/12621909/SecondaryIndex%20Design_Updated_2.pdf

Confidentiality Notice:  The information contained in this message,
including any attachments hereto, may be confidential and is intended to
be read only by the individual or entity to whom this message is
addressed. If the reader of this message is not the intended recipient or
an agent or designee of the intended recipient, please note that any
review, use, disclosure or distribution of this message or its
attachments, in any form, is strictly prohibited.  If you have received
this message in error, please immediately notify the sender and/or
[EMAIL PROTECTED] and delete or destroy any copy of this message
and its attachments.
Confidentiality Notice:  The information contained in this message,
including any attachments hereto, may be confidential and is intended to
be read only by the individual or entity to whom this message is
addressed. If the reader of this message is not the intended recipient or
an agent or designee of the intended recipient, please note that any
review, use, disclosure or distribution of this message or its
attachments, in any form, is strictly prohibited.  If you have received
this message in error, please immediately notify the sender and/or
[EMAIL PROTECTED] and delete or destroy any copy of this message
and its attachments.