Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase, mail # dev - Retry HTable.put() on client-side to handle temp connectivity problem


+
Alex Baranau 2011-06-27, 20:08
+
Ted Yu 2011-06-27, 20:16
+
Alex Baranau 2011-06-27, 20:33
+
Stack 2011-06-27, 20:40
+
Alex Baranau 2011-06-27, 20:56
+
Stack 2011-06-27, 21:21
Copy link to this message
-
Re: Retry HTable.put() on client-side to handle temp connectivity problem
Joey Echeverria 2011-06-27, 22:23
If I could override the default, I'd be a hesitant +1. I'd rather see
the default be something like retry 10 times, then throw an error.
With one option being infinite retries.

-Joey

On Mon, Jun 27, 2011 at 2:21 PM, Stack <[EMAIL PROTECTED]> wrote:
> I'd be fine with changing the default in hbase so clients just keep
> trying.  What do others think?
> St.Ack
>
> On Mon, Jun 27, 2011 at 1:56 PM, Alex Baranau <[EMAIL PROTECTED]> wrote:
>> The code I pasted works for me: it reconnects successfully. Just thought it
>> might be not the best way to do it.. I realized that by using HBase
>> configuration properties we could just say that it's up to user to configure
>> HBase client (created by Flume) properly (e.g. by adding hbase-site.xml with
>> settings to classpath). On the other hand, it looks to me that users of
>> HBase sinks will *always* want it to retry writing to HBase until it works
>> out. But default configuration works not this way: sinks stops when HBase is
>> temporarily down or inaccessible. Hence it makes using the sink more
>> complicated (because default configuration sucks), which I'd like to avoid
>> here by adding the code above. Ideally the default configuration should work
>> the best way for general-purpose case.
>>
>> I understood what are the ways to implement/configure such behavior. I think
>> we should discuss what is the best default behavior and do we need to allow
>> user override it on Flume ML (or directly at
>> https://issues.cloudera.org/browse/FLUME-685).
>>
>> Thank you guys,
>>
>> Alex Baranau
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
>>
>>
>> On Mon, Jun 27, 2011 at 11:40 PM, Stack <[EMAIL PROTECTED]> wrote:
>>
>>> Either should work Alex.  Your version will go "for ever".  Have you
>>> tried yanking hbase out from under the client to see if it reconnects?
>>>
>>> Good on you,
>>> St.Ack
>>>
>>> On Mon, Jun 27, 2011 at 1:33 PM, Alex Baranau <[EMAIL PROTECTED]>
>>> wrote:
>>> > Yes, that is what intended, I think. To make the whole picture clear,
>>> here's
>>> > the context:
>>> >
>>> > * there's a Flume's HBase sink (read: HBase client) which writes data
>>> from
>>> > Flume "pipe" (read: some event-based messages source) to HTable;
>>> > * when HBase is down for some time (with default HBase configuration on
>>> > Flume's sink side) HTable.put throws exception and client exits (it
>>> usually
>>> > takes ~10 min to fail);
>>> > * Flume is smart enough to accumulate data to be written reliably if sink
>>> > behaves badly (not writing for some time, pauses, etc.), so it would be
>>> > great if the sink tries to write data until HBase is up again, BUT:
>>> > * but here, as we have complete "failure" of sink process (thread needs
>>> to
>>> > be restarted) the data never reaches HTable even after HBase cluster is
>>> > brought up again.
>>> >
>>> > So you suggest instead of this extra construction around HTable.put to
>>> use
>>> > configuration properties "hbase.client.pause" and
>>> > "hbase.client.retries.number"? I.e. make retries attempts to be
>>> (reasonably)
>>> > close to "perform forever". Is that what you meant?
>>> >
>>> > Thank you,
>>> > Alex Baranau
>>> > ----
>>> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
>>> HBase
>>> >
>>> > On Mon, Jun 27, 2011 at 11:16 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> >
>>> >> This would retry indefinitely, right ?
>>> >> Normally maximum retry duration would govern how long the retry is
>>> >> attempted.
>>> >>
>>> >> On Mon, Jun 27, 2011 at 1:08 PM, Alex Baranau <[EMAIL PROTECTED]
>>> >> >wrote:
>>> >>
>>> >> > Hello,
>>> >> >
>>> >> > Just wanted to confirm that I'm doing things in a proper way here. How
>>> >> > about
>>> >> > this code to handle the temp cluster connectivity problems (or cluster
>>> >> down
>>> >> > time) on client-side?
>>> >> >
>>> >> > +    // HTable.put() will fail with exception if connection to cluster
>>> is
>>> >> > temporarily broken or

Joseph Echeverria
Cloudera, Inc.
443.305.9434
+
Gary Helmling 2011-06-28, 16:17
+
Doug Meil 2011-06-28, 16:40
+
Alex Baranau 2011-06-28, 17:07
+
Doug Meil 2011-06-28, 19:41
+
Alex Baranau 2011-06-29, 06:17
+
Doug Meil 2011-06-29, 13:44
+
Alex Baranau 2011-06-29, 14:57
+
Andrew Purtell 2011-06-28, 23:17
+
Todd Lipcon 2011-06-28, 15:45