Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Connection reference counting


Copy link to this message
-
Re: Connection reference counting
;-)

For backward compatibility it would be better to keep the existing
behaviour and add a method, but I by default I would be lazy here and just
fix a behaviour that seems broken.
On Wed, Jul 17, 2013 at 7:06 PM, Lars George <[EMAIL PROTECTED]> wrote:

> Hi Nicholas,
>
> Knock yourself out :)
>
> I would either call deleteConnection() with the force flag set to true
> (like deleteStaleConnection does) or check if the connection is closed -
> using isClosed() - and only if it is remove it from the internal list.
>
> Best would be to expose the force flag to the users with an overloaded
> deleteAllConnections() plus the above part of not removing not closed
> connections.
>
> Lars
>
>
>
> On Jul 17, 2013, at 6:31 PM, Nicolas Liochon <[EMAIL PROTECTED]> wrote:
>
> > I had a look, I think it's a bug.
> > If the connection holds resources that should be freed, well they won't
> be
> > freed.
> > It's used in the minicluster shutdown. So we could imagine scenarios
> which
> > that create some test flakyness. I doubt it happens in real life, but
> imho
> > it's better to fix it (it there are no different opinions, I will do it
> if
> > you don't ;-).
> >
> > Cheers,
> >
> > Nicolas
> >
> >
> > On Wed, Jul 17, 2013 at 10:03 AM, Lars George <[EMAIL PROTECTED]>
> wrote:
> >
> >> Hi,
> >>
> >> I am working on an issue around Thrift 2 (HBASE-7035), and I am testing
> >> how this all works a bit. See
> >> https://github.com/larsgeorge/hbase-scanner-test for the code, which
> runs
> >> against a local HBase instance. Now, there is an issue that connections
> do
> >> not get closed as I would have expected. I close the connection in a
> loop
> >> over a scanner that gets single rows. Although I close it, it keeps on
> >> going. Scanners may or may not be reference counted - any code calling
> >> HCM.getConnection(conf) increases the count.
> >>
> >> So I tried a HCM.deleteAllConnections, which does this:
> >>
> >>  public static void deleteAllConnections() {
> >>    synchronized (HBASE_INSTANCES) {
> >>      Set<HConnectionKey> connectionKeys = new HashSet<HConnectionKey>();
> >>      connectionKeys.addAll(HBASE_INSTANCES.keySet());
> >>      for (HConnectionKey connectionKey : connectionKeys) {
> >>        deleteConnection(connectionKey, false);
> >>      }
> >>      HBASE_INSTANCES.clear();
> >>    }
> >>  }
> >>
> >> It iterates over all connections and "deletes" them. Then is clears the
> >> entire reference list! The issue is that deleteConnection() is using the
> >> refcounts and is *not* closing the connection when it is still
> referenced.
> >> The final clear() call simply drops them from the list of managed
> >> connections. This means that we now might have dangling open
> connections,
> >> and unless you hold a reference there is no way that you can talk to it
> >> again, not even using deleteStaleConnection() for example.
> >>
> >> Is that wanted? Just curious, since obviously this is not a biggie in
> real
> >> life.
> >>
> >> Lars
> >>
> >>
> >>
> >>
> >>
> >>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB