Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Client Performance Bottleneck in a Single Virtual Machine


Copy link to this message
-
Re: HBase Client Performance Bottleneck in a Single Virtual Machine
He uses HConnection.getTable()   which in turn uses the Htable constructor
HTable(*final* *byte*[] tableName, *final* HConnection connection,
*final*ExecutorService pool)

So no worry. on HTable#close() the connection wont get closed   :)

-Anoop-
On Mon, Nov 4, 2013 at 11:29 AM, Sriram Ramachandrasekaran <
[EMAIL PROTECTED]> wrote:

> HTable is the implementation of HTableInterface. I was looking at the code
> and it *indeed* closes the underlying resources on close() unless you
> create it with the ExecutorService and HConnection Option that lars
> suggested. Please do take a look at HTable constructors - that might help.
>
> P.S: I verified this on 0.94.6 code base. Hope things haven't changed b/w
> this and your version (0.94.12)
>
>
>
> On Mon, Nov 4, 2013 at 11:21 AM, <[EMAIL PROTECTED]> wrote:
>
> > Our current usage is how I would do this in a typical database app with
> > table acting like a statement. It looks like this:
> >
> > Connection connection = null;
> > HTableInterface table = null;
> > try {
> >         connection = pool.acquire();
> >         table = connection.getTable(tableName);
> >         // Do work
> > } finally {
> >         table.close();
> >         pool.release(connection);
> > }
> >
> > Is this incorrect? The API docs says close " Releases any resources held
> > or pending changes in internal buffers." I didn't interpret that as
> having
> > it close the underlying connection. Thanks!
> >
> > -Mike
> >
> > -----Original Message-----
> > From: Sriram Ramachandrasekaran [mailto:[EMAIL PROTECTED]]
> > Sent: Sunday, November 03, 2013 11:43 PM
> > To: [EMAIL PROTECTED]
> > Cc: [EMAIL PROTECTED]
> > Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> > Machine
> >
> > Hey Michael - Per API documentation, closing the HTable Instance would
> > close the underlying resources too. Hope you are aware of it.
> >
> >
> > On Mon, Nov 4, 2013 at 11:06 AM, <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi Lars, at application startup the pool is created with X number of
> > > connections using the first method you indicated:
> > > HConnectionManager.createConnection(conf). We store each connection in
> > > the pool automatically and serve it up to threads as they request it.
> > > When a thread is done using the connection, they return it back to the
> > > pool. The connections are not be created and closed per thread, but
> > > only once for the entire application. We are using the
> > > GenericObjectPool from Apache Commons Pooling as the foundation of our
> > > connection pooling approach. Our entire pool implementation really
> > > consists of just a couple overridden methods to specify how to create
> > > a new connection and close it. The GenericObjectPool class does all the
> > rest. See here for details:
> > > http://commons.apache.org/proper/commons-pool/
> > >
> > > Each thread is getting a HTableInstance as needed and then closing it
> > > when done. The only thing we are not doing is using the
> > > createConnection method that takes in an ExecutorService as that
> > > wouldn't work in our model. Our app is like a web application - the
> > > thread pool is managed outside the scope of our application code so we
> > > can't assume the service is available at connection creation time.
> > Thanks!
> > >
> > > -Mike
> > >
> > >
> > > -----Original Message-----
> > > From: lars hofhansl [mailto:[EMAIL PROTECTED]]
> > > Sent: Sunday, November 03, 2013 11:27 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> > > Machine
> > >
> > > Hi Micheal,
> > >
> > > can you try to create a single HConnection in your client:
> > > HConnectionManager.createConnection(Configuration conf) or
> > > HConnectionManager.createConnection(Configuration conf,
> > > ExecutorService
> > > pool)
> > >
> > > Then use HConnection.getTable(...) each time you need to do an
> operation.
> > >
> > > I.e.
> > > Configuration conf = ...;
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB