Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Client Performance Bottleneck in a Single Virtual Machine


Copy link to this message
-
Re: HBase Client Performance Bottleneck in a Single Virtual Machine
Anoop John 2013-11-04, 05:58
If you have used con.getTable() the close on HTable wont close the
underlying connection

-Anoop-

On Mon, Nov 4, 2013 at 11:21 AM, <[EMAIL PROTECTED]> wrote:

> Our current usage is how I would do this in a typical database app with
> table acting like a statement. It looks like this:
>
> Connection connection = null;
> HTableInterface table = null;
> try {
>         connection = pool.acquire();
>         table = connection.getTable(tableName);
>         // Do work
> } finally {
>         table.close();
>         pool.release(connection);
> }
>
> Is this incorrect? The API docs says close " Releases any resources held
> or pending changes in internal buffers." I didn't interpret that as having
> it close the underlying connection. Thanks!
>
> -Mike
>
> -----Original Message-----
> From: Sriram Ramachandrasekaran [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, November 03, 2013 11:43 PM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> Machine
>
> Hey Michael - Per API documentation, closing the HTable Instance would
> close the underlying resources too. Hope you are aware of it.
>
>
> On Mon, Nov 4, 2013 at 11:06 AM, <[EMAIL PROTECTED]> wrote:
>
> > Hi Lars, at application startup the pool is created with X number of
> > connections using the first method you indicated:
> > HConnectionManager.createConnection(conf). We store each connection in
> > the pool automatically and serve it up to threads as they request it.
> > When a thread is done using the connection, they return it back to the
> > pool. The connections are not be created and closed per thread, but
> > only once for the entire application. We are using the
> > GenericObjectPool from Apache Commons Pooling as the foundation of our
> > connection pooling approach. Our entire pool implementation really
> > consists of just a couple overridden methods to specify how to create
> > a new connection and close it. The GenericObjectPool class does all the
> rest. See here for details:
> > http://commons.apache.org/proper/commons-pool/
> >
> > Each thread is getting a HTableInstance as needed and then closing it
> > when done. The only thing we are not doing is using the
> > createConnection method that takes in an ExecutorService as that
> > wouldn't work in our model. Our app is like a web application - the
> > thread pool is managed outside the scope of our application code so we
> > can't assume the service is available at connection creation time.
> Thanks!
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: lars hofhansl [mailto:[EMAIL PROTECTED]]
> > Sent: Sunday, November 03, 2013 11:27 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> > Machine
> >
> > Hi Micheal,
> >
> > can you try to create a single HConnection in your client:
> > HConnectionManager.createConnection(Configuration conf) or
> > HConnectionManager.createConnection(Configuration conf,
> > ExecutorService
> > pool)
> >
> > Then use HConnection.getTable(...) each time you need to do an operation.
> >
> > I.e.
> > Configuration conf = ...;
> > ExecutorService pool = ...;
> > // create a single HConnection for you vm.
> > HConnection con = HConnectionManager.createConnection(Configuration
> > conf, ExecutorService pool); // reuse the connection for many tables,
> > even in different threads HTableInterface table = con.getTable(...);
> > // use table even for only a few operation.
> > table.close();
> > ...
> > HTableInterface table = con.getTable(...); // use table even for only
> > a few operation.
> > table.close();
> > ...
> > // at the end close the connection
> > con.close();
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: "[EMAIL PROTECTED]"
> > <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Sunday, November 3, 2013 7:46 PM
> > Subject: HBase Client Performance Bottleneck in a Single Virtual