Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Client Performance Bottleneck in a Single Virtual Machine


Copy link to this message
-
Re: HBase Client Performance Bottleneck in a Single Virtual Machine
If you have used con.getTable() the close on HTable wont close the
underlying connection

-Anoop-

On Mon, Nov 4, 2013 at 11:21 AM, <[EMAIL PROTECTED]> wrote:

> Our current usage is how I would do this in a typical database app with
> table acting like a statement. It looks like this:
>
> Connection connection = null;
> HTableInterface table = null;
> try {
>         connection = pool.acquire();
>         table = connection.getTable(tableName);
>         // Do work
> } finally {
>         table.close();
>         pool.release(connection);
> }
>
> Is this incorrect? The API docs says close " Releases any resources held
> or pending changes in internal buffers." I didn't interpret that as having
> it close the underlying connection. Thanks!
>
> -Mike
>
> -----Original Message-----
> From: Sriram Ramachandrasekaran [mailto:[EMAIL PROTECTED]]
> Sent: Sunday, November 03, 2013 11:43 PM
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> Machine
>
> Hey Michael - Per API documentation, closing the HTable Instance would
> close the underlying resources too. Hope you are aware of it.
>
>
> On Mon, Nov 4, 2013 at 11:06 AM, <[EMAIL PROTECTED]> wrote:
>
> > Hi Lars, at application startup the pool is created with X number of
> > connections using the first method you indicated:
> > HConnectionManager.createConnection(conf). We store each connection in
> > the pool automatically and serve it up to threads as they request it.
> > When a thread is done using the connection, they return it back to the
> > pool. The connections are not be created and closed per thread, but
> > only once for the entire application. We are using the
> > GenericObjectPool from Apache Commons Pooling as the foundation of our
> > connection pooling approach. Our entire pool implementation really
> > consists of just a couple overridden methods to specify how to create
> > a new connection and close it. The GenericObjectPool class does all the
> rest. See here for details:
> > http://commons.apache.org/proper/commons-pool/
> >
> > Each thread is getting a HTableInstance as needed and then closing it
> > when done. The only thing we are not doing is using the
> > createConnection method that takes in an ExecutorService as that
> > wouldn't work in our model. Our app is like a web application - the
> > thread pool is managed outside the scope of our application code so we
> > can't assume the service is available at connection creation time.
> Thanks!
> >
> > -Mike
> >
> >
> > -----Original Message-----
> > From: lars hofhansl [mailto:[EMAIL PROTECTED]]
> > Sent: Sunday, November 03, 2013 11:27 PM
> > To: [EMAIL PROTECTED]
> > Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> > Machine
> >
> > Hi Micheal,
> >
> > can you try to create a single HConnection in your client:
> > HConnectionManager.createConnection(Configuration conf) or
> > HConnectionManager.createConnection(Configuration conf,
> > ExecutorService
> > pool)
> >
> > Then use HConnection.getTable(...) each time you need to do an operation.
> >
> > I.e.
> > Configuration conf = ...;
> > ExecutorService pool = ...;
> > // create a single HConnection for you vm.
> > HConnection con = HConnectionManager.createConnection(Configuration
> > conf, ExecutorService pool); // reuse the connection for many tables,
> > even in different threads HTableInterface table = con.getTable(...);
> > // use table even for only a few operation.
> > table.close();
> > ...
> > HTableInterface table = con.getTable(...); // use table even for only
> > a few operation.
> > table.close();
> > ...
> > // at the end close the connection
> > con.close();
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> >  From: "[EMAIL PROTECTED]"
> > <[EMAIL PROTECTED]>
> > To: [EMAIL PROTECTED]
> > Sent: Sunday, November 3, 2013 7:46 PM
> > Subject: HBase Client Performance Bottleneck in a Single Virtual
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB