Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> zookeeper-3.3.2 has default maxClientCnxns set to 10?


+
Alex Baranau 2011-03-25, 19:36
+
Stack 2011-03-25, 20:00
+
Alex Baranau 2011-03-25, 20:57
+
Mathias Herberts 2011-03-25, 21:22
+
Alex Baranau 2011-03-25, 22:21
+
Todd Lipcon 2011-03-26, 01:07
+
Dmitriy Lyubimov 2011-03-26, 09:04
+
Dmitriy Lyubimov 2011-03-26, 09:08
Copy link to this message
-
Re: zookeeper-3.3.2 has default maxClientCnxns set to 10?
On Sat, Mar 26, 2011 at 2:08 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]> wrote:

> or perhaps scratch that. it seems you are saying the problem arises on
> the backend and your reducer code certainly doesn't create over 10
> connections there. so it might be a combination of other tasks running
> at the same address.
>

Yes, in my case it's a matter of machines with 20+ MR slots, each of which
consumes a ZK connection.

-Todd
>
> On Sat, Mar 26, 2011 at 2:04 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]>
> wrote:
> > yes i had a very similar  issue although i prefer to think about it in
> > terms of hbase and by extension zk connection leak in TableInputFormat
> > rather than adjusting max zk connection to 30 'cause sooner or later
> > you will run out of it too.
> >
> > The problem is that HConnectionManager now identifies hbase connection
> > by identity comparison of the Configuration object. Which is kind of
> > unfortunate because cloning configuration is a common practice and in
> > reality it means every MR client would open its own connection to
> > hbase.
> >
> > To exacerbate situation, TableInput/OutputFormats do not relinquish
> > unshared connection and there's very little you can do work around
> > this. I posted workaround to this i used with cdh3b4 just a day or two
> > ago. The issue turned out to be a pain as it really quite hard to
> > figure where and whether configuration objects get cloned or not.
> >
> > Also, if you use HTable, be mindful that you may want to release that
> > connection explicitly too by calling
> > HConnectionManager.deleteConnection (conf, true).
> >
> > On Fri, Mar 25, 2011 at 1:57 PM, Alex Baranau <[EMAIL PROTECTED]>
> wrote:
> >> I see what you are asking. I'm using stand-alone Zookeeper, not
> "internal"
> >> one of HBase. So it reads configuration only form zoo.cfg. And it seems
> that
> >> by default (when maxClientCnxns is absent in it) it acts like
> >> maxClientCnxns=10. I'd expect it to be unlimited when this property is
> >> omitted. At least it was like this..
> >>
> >> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users
> won't
> >> like it. What do you guys think?
> >>
> >> Alex Baranau
> >> ----
> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
> HBase
> >> Hadoop ecosystem search :: http://search-hadoop.com/
> >>
> >> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
> >>
> >>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <
> [EMAIL PROTECTED]>
> >>> wrote:
> >>> > As far as I know HBase configured to initiate up to 30 connections by
> >>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
> >>>
> >>> Yes
> >>>
> >>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
> >>> for zk?).  Is it possible its not reading hbase-site.xml when the task
> >>> runs?
> >>>
> >>> St.Ack
> >>>
> >>
> >
>

--
Todd Lipcon
Software Engineer, Cloudera