Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Coprocessor Increments


Copy link to this message
-
Re: Coprocessor Increments
Inline.

On Sun, Oct 13, 2013 at 6:02 AM, Michael Segel <[EMAIL PROTECTED]>wrote:

> Ok…
>
> Sure you can have your app update the secondary index table.
> The only issue with that is if someone updates the base table outside of
> your app,
> they may or may not increment the secondary index.
>
Anil: We dont allow people to write data into HBase from their own HBase
client. We control the writes into HBase. So, we dont have the problem of
secondary index not getting written.
For example, If you expose a restful web service you can easily control the
writes to HBase. Even, if user requests to write one row in "main table",
you application can have the logic to writing in "Secondary index" tables.
In this way, it is transparent to users also. You can add/remove seconday
indexes as you want.

> Note that your secondary index doesn't have to be an inverted table, but
> could be SOLR, LUCENE or something else.
>
Anil:As of now, we are happy with Inverted tables as they fit to our use
case.

>
> So you really want to secondary indexes on the server.
>
> There are a couple of things that could improve the performance, although
> the write to the secondary index would most likely lag under heavy load.
>
>
> On Oct 12, 2013, at 11:27 PM, anil gupta <[EMAIL PROTECTED]> wrote:
>
> > John,
> >
> > My 2 cents:
> > I tried implementing Secondary Index by using Region Observers on Put. It
> > works well under low load. But, under heavy load the RO could not keep up
> > with load cross region server writes.
> > Then, i decided not to use RO as per Andrew's explanation and  I moved
> all
> > the logic of building secondary index tables on my HBase Client . Since
> > then, the system has been running fine under heavy load.
> > IMO, if you will use RO and do cross RS read/write then perhaps this will
> > become your bottleneck in HBase.
> > Is it possible for you to avoid RO and control the writes/updates from
> > client side?
> >
> > Thanks,
> > Anil Gupta
> >
> >
> > On Fri, Oct 11, 2013 at 6:06 PM, John Weatherford <
> > [EMAIL PROTECTED]> wrote:
> >
> >> OP Here :)
> >>
> >> Our current design involves a Region Observer on a table that does
> >> increments on a second table. We took the approach that Michael said and
> >> inside the RO, we got a new connection and everything. We believe this
> is
> >> causing deadlocks for us. Our next attempt is going to be writing to
> >> another row in the same table where we will store the increments. If
> this
> >> doesn't work, we are going to simply pull the increments out of the RO
> and
> >> do them in the application or in Flume.
> >>
> >> @Tom Brown
> >> I would be very interested to hear more about your solution of
> >> aggregating the increments in another system that is then responsible
> for
> >> updating in Hbase.
> >>
> >> -jW
> >>
> >>
> >> On Fri 11 Oct 2013 10:26:58 AM PDT, Vladimir Rodionov wrote:
> >>
> >>> With respect to the OP's design… does the deadlock occur because he's
> >>>>> trying to update a column in a different row within the same table?
> >>>>>
> >>>>
> >>> Because he is trying to update *row* in a different Region (and
> >>> potentially in different RS).
> >>>
> >>> Best regards,
> >>> Vladimir Rodionov
> >>> Principal Platform Engineer
> >>> Carrier IQ, www.carrieriq.com
> >>> e-mail: [EMAIL PROTECTED]
> >>>
> >>> ______________________________**__________
> >>> From: Michael Segel [[EMAIL PROTECTED]]
> >>> Sent: Friday, October 11, 2013 9:10 AM
> >>> To: [EMAIL PROTECTED]
> >>> Cc: Vladimir Rodionov
> >>> Subject: Re: Coprocessor Increments
> >>>
> >>>
> >>> Confidentiality Notice:  The information contained in this message,
> >>> including any attachments hereto, may be confidential and is intended
> to be
> >>> read only by the individual or entity to whom this message is
> addressed. If
> >>> the reader of this message is not the intended recipient or an agent or
> >>> designee of the intended recipient, please note that any review, use,
Thanks & Regards,
Anil Gupta