Hello,

I've set up a test HBase+Hadoop cluster yesterday and got the following
error in logs during running MR job (which internally creates HTable for
Reducer):

KeeperErrorCode = ConnectionLoss for /hbase

Then I went to Zookeeper logs and found this:

2011-03-24 22:41:49,884 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/x.x.x.x - max is 10

As far as I know HBase configured to initiate up to 30 connections by
default, and maxClientCnxns for Zookeeper was meant to be 30 as well. Is
this a bug that I have it set to 10 by default? I use a version:
zookeeper-3.3.2-CDH3B4. Sorry if this issue is specific to CDH (then I
probably should post it in the correspondent ML).

After setting maxClientCnxns explicitly to 30 in zoo.cfg things work well.

Just to prevent some Qs, I have this in code:

when starting job:
    Configuration conf = HBaseConfiguration.create();
    Job job = new Job(conf, aggRawDataJobClass.getName());
    TableMapReduceUtil.initTableMapperJob(rawLogsTableName, scan,
            mapperClass, ImmutableBytesWritable.class, mappedAggregateClass,
job);

in reducer:
public static class CustomReducer<T> extends
TableReducer<ImmutableBytesWritable, T, Writable> {
    @Override
    protected void setup(Context context) throws IOException,
InterruptedException {
       ...
       hTable = new HTable(context.getConfiguration(), tableName);
       ...
    }
}

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/

On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <[EMAIL PROTECTED]> wrote:
> As far as I know HBase configured to initiate up to 30 connections by
> default, and maxClientCnxns for Zookeeper was meant to be 30 as well.

Yes

I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
for zk?).  Is it possible its not reading hbase-site.xml when the task
runs?

St.Ack

I see what you are asking. I'm using stand-alone Zookeeper, not "internal"
one of HBase. So it reads configuration only form zoo.cfg. And it seems that
by default (when maxClientCnxns is absent in it) it acts like
maxClientCnxns=10. I'd expect it to be unlimited when this property is
omitted. At least it was like this..

Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users won't
like it. What do you guys think?

Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/

On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <[EMAIL PROTECTED]>
> wrote:
> > As far as I know HBase configured to initiate up to 30 connections by
> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
>
> Yes
>
> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
> for zk?).  Is it possible its not reading hbase-site.xml when the task
> runs?
>
> St.Ack
>

It's 10 connections from the same host, I thought HBase was using a
wrapper to optimize the number of ZK clients?

On Fri, Mar 25, 2011 at 21:57, Alex Baranau <[EMAIL PROTECTED]> wrote:
> I see what you are asking. I'm using stand-alone Zookeeper, not "internal"
> one of HBase. So it reads configuration only form zoo.cfg. And it seems that
> by default (when maxClientCnxns is absent in it) it acts like
> maxClientCnxns=10. I'd expect it to be unlimited when this property is
> omitted. At least it was like this..
>
> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users won't
> like it. What do you guys think?
>
> Alex Baranau
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
> Hadoop ecosystem search :: http://search-hadoop.com/
>
> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <[EMAIL PROTECTED]>
>> wrote:
>> > As far as I know HBase configured to initiate up to 30 connections by
>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
>>
>> Yes
>>
>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
>> for zk?).  Is it possible its not reading hbase-site.xml when the task
>> runs?
>>
>> St.Ack
>>
>

Right, from the same host (same ip). But in HBase I think the default max
number of connections is set to 30. Please correct me if I'm wrong. If I'm
right, then we should probably change either of the defaults. No?
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/

On Fri, Mar 25, 2011 at 5:22 PM, Mathias Herberts <
[EMAIL PROTECTED]> wrote:

> It's 10 connections from the same host, I thought HBase was using a
> wrapper to optimize the number of ZK clients?
>
> On Fri, Mar 25, 2011 at 21:57, Alex Baranau <[EMAIL PROTECTED]>
> wrote:
> > I see what you are asking. I'm using stand-alone Zookeeper, not
> "internal"
> > one of HBase. So it reads configuration only form zoo.cfg. And it seems
> that
> > by default (when maxClientCnxns is absent in it) it acts like
> > maxClientCnxns=10. I'd expect it to be unlimited when this property is
> > omitted. At least it was like this..
> >
> > Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users
> won't
> > like it. What do you guys think?
> >
> > Alex Baranau
> > ----
> > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
> HBase
> > Hadoop ecosystem search :: http://search-hadoop.com/
> >
> > On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
> >
> >> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <
> [EMAIL PROTECTED]>
> >> wrote:
> >> > As far as I know HBase configured to initiate up to 30 connections by
> >> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
> >>
> >> Yes
> >>
> >> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
> >> for zk?).  Is it possible its not reading hbase-site.xml when the task
> >> runs?
> >>
> >> St.Ack
> >>
> >
>

I came upon this independently today, actually. Filed ZOOKEEPER-1030

On Fri, Mar 25, 2011 at 3:21 PM, Alex Baranau <[EMAIL PROTECTED]>wrote:

> Right, from the same host (same ip). But in HBase I think the default max
> number of connections is set to 30. Please correct me if I'm wrong. If I'm
> right, then we should probably change either of the defaults. No?
>
>
> Alex Baranau
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
> Hadoop ecosystem search :: http://search-hadoop.com/
>
> On Fri, Mar 25, 2011 at 5:22 PM, Mathias Herberts <
> [EMAIL PROTECTED]> wrote:
>
> > It's 10 connections from the same host, I thought HBase was using a
> > wrapper to optimize the number of ZK clients?
> >
> > On Fri, Mar 25, 2011 at 21:57, Alex Baranau <[EMAIL PROTECTED]>
> > wrote:
> > > I see what you are asking. I'm using stand-alone Zookeeper, not
> > "internal"
> > > one of HBase. So it reads configuration only form zoo.cfg. And it seems
> > that
> > > by default (when maxClientCnxns is absent in it) it acts like
> > > maxClientCnxns=10. I'd expect it to be unlimited when this property is
> > > omitted. At least it was like this..
> > >
> > > Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users
> > won't
> > > like it. What do you guys think?
> > >
> > > Alex Baranau
> > > ----
> > > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
> > HBase
> > > Hadoop ecosystem search :: http://search-hadoop.com/
> > >
> > > On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
> > >
> > >> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <
> > [EMAIL PROTECTED]>
> > >> wrote:
> > >> > As far as I know HBase configured to initiate up to 30 connections
> by
> > >> > default, and maxClientCnxns for Zookeeper was meant to be 30 as
> well.
> > >>
> > >> Yes
> > >>
> > >> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
> > >> for zk?).  Is it possible its not reading hbase-site.xml when the task
> > >> runs?
> > >>
> > >> St.Ack
> > >>
> > >
> >
>

--
Todd Lipcon
Software Engineer, Cloudera

yes i had a very similar  issue although i prefer to think about it in
terms of hbase and by extension zk connection leak in TableInputFormat
rather than adjusting max zk connection to 30 'cause sooner or later
you will run out of it too.

The problem is that HConnectionManager now identifies hbase connection
by identity comparison of the Configuration object. Which is kind of
unfortunate because cloning configuration is a common practice and in
reality it means every MR client would open its own connection to
hbase.

To exacerbate situation, TableInput/OutputFormats do not relinquish
unshared connection and there's very little you can do work around
this. I posted workaround to this i used with cdh3b4 just a day or two
ago. The issue turned out to be a pain as it really quite hard to
figure where and whether configuration objects get cloned or not.

Also, if you use HTable, be mindful that you may want to release that
connection explicitly too by calling
HConnectionManager.deleteConnection (conf, true).

On Fri, Mar 25, 2011 at 1:57 PM, Alex Baranau <[EMAIL PROTECTED]> wrote:
> I see what you are asking. I'm using stand-alone Zookeeper, not "internal"
> one of HBase. So it reads configuration only form zoo.cfg. And it seems that
> by default (when maxClientCnxns is absent in it) it acts like
> maxClientCnxns=10. I'd expect it to be unlimited when this property is
> omitted. At least it was like this..
>
> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users won't
> like it. What do you guys think?
>
> Alex Baranau
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
> Hadoop ecosystem search :: http://search-hadoop.com/
>
> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <[EMAIL PROTECTED]>
>> wrote:
>> > As far as I know HBase configured to initiate up to 30 connections by
>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
>>
>> Yes
>>
>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
>> for zk?).  Is it possible its not reading hbase-site.xml when the task
>> runs?
>>
>> St.Ack
>>
>

or perhaps scratch that. it seems you are saying the problem arises on
the backend and your reducer code certainly doesn't create over 10
connections there. so it might be a combination of other tasks running
at the same address.

On Sat, Mar 26, 2011 at 2:04 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]> wrote:
> yes i had a very similar  issue although i prefer to think about it in
> terms of hbase and by extension zk connection leak in TableInputFormat
> rather than adjusting max zk connection to 30 'cause sooner or later
> you will run out of it too.
>
> The problem is that HConnectionManager now identifies hbase connection
> by identity comparison of the Configuration object. Which is kind of
> unfortunate because cloning configuration is a common practice and in
> reality it means every MR client would open its own connection to
> hbase.
>
> To exacerbate situation, TableInput/OutputFormats do not relinquish
> unshared connection and there's very little you can do work around
> this. I posted workaround to this i used with cdh3b4 just a day or two
> ago. The issue turned out to be a pain as it really quite hard to
> figure where and whether configuration objects get cloned or not.
>
> Also, if you use HTable, be mindful that you may want to release that
> connection explicitly too by calling
> HConnectionManager.deleteConnection (conf, true).
>
> On Fri, Mar 25, 2011 at 1:57 PM, Alex Baranau <[EMAIL PROTECTED]> wrote:
>> I see what you are asking. I'm using stand-alone Zookeeper, not "internal"
>> one of HBase. So it reads configuration only form zoo.cfg. And it seems that
>> by default (when maxClientCnxns is absent in it) it acts like
>> maxClientCnxns=10. I'd expect it to be unlimited when this property is
>> omitted. At least it was like this..
>>
>> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users won't
>> like it. What do you guys think?
>>
>> Alex Baranau
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
>> Hadoop ecosystem search :: http://search-hadoop.com/
>>
>> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
>>
>>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <[EMAIL PROTECTED]>
>>> wrote:
>>> > As far as I know HBase configured to initiate up to 30 connections by
>>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
>>>
>>> Yes
>>>
>>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
>>> for zk?).  Is it possible its not reading hbase-site.xml when the task
>>> runs?
>>>
>>> St.Ack
>>>
>>
>

On Sat, Mar 26, 2011 at 2:08 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]> wrote:

> or perhaps scratch that. it seems you are saying the problem arises on
> the backend and your reducer code certainly doesn't create over 10
> connections there. so it might be a combination of other tasks running
> at the same address.
>

Yes, in my case it's a matter of machines with 20+ MR slots, each of which
consumes a ZK connection.

-Todd
>
> On Sat, Mar 26, 2011 at 2:04 AM, Dmitriy Lyubimov <[EMAIL PROTECTED]>
> wrote:
> > yes i had a very similar  issue although i prefer to think about it in
> > terms of hbase and by extension zk connection leak in TableInputFormat
> > rather than adjusting max zk connection to 30 'cause sooner or later
> > you will run out of it too.
> >
> > The problem is that HConnectionManager now identifies hbase connection
> > by identity comparison of the Configuration object. Which is kind of
> > unfortunate because cloning configuration is a common practice and in
> > reality it means every MR client would open its own connection to
> > hbase.
> >
> > To exacerbate situation, TableInput/OutputFormats do not relinquish
> > unshared connection and there's very little you can do work around
> > this. I posted workaround to this i used with cdh3b4 just a day or two
> > ago. The issue turned out to be a pain as it really quite hard to
> > figure where and whether configuration objects get cloned or not.
> >
> > Also, if you use HTable, be mindful that you may want to release that
> > connection explicitly too by calling
> > HConnectionManager.deleteConnection (conf, true).
> >
> > On Fri, Mar 25, 2011 at 1:57 PM, Alex Baranau <[EMAIL PROTECTED]>
> wrote:
> >> I see what you are asking. I'm using stand-alone Zookeeper, not
> "internal"
> >> one of HBase. So it reads configuration only form zoo.cfg. And it seems
> that
> >> by default (when maxClientCnxns is absent in it) it acts like
> >> maxClientCnxns=10. I'd expect it to be unlimited when this property is
> >> omitted. At least it was like this..
> >>
> >> Aha! I see they changed default to 10 *in 3.3.x*. I think HBase users
> won't
> >> like it. What do you guys think?
> >>
> >> Alex Baranau
> >> ----
> >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
> HBase
> >> Hadoop ecosystem search :: http://search-hadoop.com/
> >>
> >> On Fri, Mar 25, 2011 at 4:00 PM, Stack <[EMAIL PROTECTED]> wrote:
> >>
> >>> On Fri, Mar 25, 2011 at 12:36 PM, Alex Baranau <
> [EMAIL PROTECTED]>
> >>> wrote:
> >>> > As far as I know HBase configured to initiate up to 30 connections by
> >>> > default, and maxClientCnxns for Zookeeper was meant to be 30 as well.
> >>>
> >>> Yes
> >>>
> >>> I'm not sure how it'd go from 30 to 10 (Is 10 the default connections
> >>> for zk?).  Is it possible its not reading hbase-site.xml when the task
> >>> runs?
> >>>
> >>> St.Ack
> >>>
> >>
> >
>

--
Todd Lipcon
Software Engineer, Cloudera