Alex Baranau 2011-03-25, 19:36
Stack 2011-03-25, 20:00
Alex Baranau 2011-03-25, 20:57
Mathias Herberts 2011-03-25, 21:22
Alex Baranau 2011-03-25, 22:21
Todd Lipcon 2011-03-26, 01:07
Dmitriy Lyubimov 2011-03-26, 09:04
Dmitriy Lyubimov 2011-03-26, 09:08
On Sat, Mar 26, 2011 at 2:08 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]> wrote:
> or perhaps scratch that. it seems you are saying the problem arises on
> the backend and your reducer code certainly doesn't create over 10
> connections there. so it might be a combination of other tasks running
> at the same address.
Yes, in my case it's a matter of machines with 20+ MR slots, each of which
consumes a ZK connection.
> On Sat, Mar 26, 2011 at 2:04 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]>
> > yes i had a very similar issue although i prefer to think about it in
> > terms of hbase and by extension zk connection leak in TableInputFormat
> > rather than adjusting max zk connection to 30 'cause sooner or later
> > you will run out of it too.
> > The problem is that HConnectionManager now identifies hbase connection
> > by identity comparison of the Configuration object. Which is kind of
> > unfortunate because cloning configuration is a common practice and in
> > reality it means every MR client would open its own connection to
> > hbase.
> > To exacerbate situation, TableInput/OutputFormats do not relinquish
> > unshared connection and there's very little you can do work around
> > this. I posted workaround to this i used with cdh3b4 just a day or two
> > ago. The issue turned out to be a pain as it really quite hard to
> > figure where and whether configuration objects get cloned or not.
> > Also, if you use HTable, be mindful that you may want to release that
> > connection explicitly too by calling
> > HConnectionManager.deleteConnection (conf, true).
> > On Fri, Mar 25, 2011 at 1:57 PM, Alex Baranau <[EMAIL PROTECTED]>
> >> I see what you are asking. I'm using stand-alone Zookeeper, not
> >> one of HBase. So it reads configuration only form zoo.cfg. And it seems
> >> by default (when maxClientCnxns is absent in it) it acts like
> >> maxClientCnxns=10. I'd expect it to be unlimited when this property is
> >> omitted. At least it was like this..
> >> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users
> >> like it. What do you guys think?
> >> Alex Baranau
> >> ----
> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
> >> Hadoop ecosystem search :: http://search-hadoop.com/
> >> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
> >>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <
> [EMAIL PROTECTED]>
> >>> wrote:
> >>> > As far as I know HBase configured to initiate up to 30 connections by
> >>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
> >>> Yes
> >>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
> >>> for zk?). Is it possible its not reading hbase-site.xml when the task
> >>> runs?
> >>> St.Ack
Software Engineer, Cloudera