Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase Client Performance Bottleneck in a Single Virtual Machine


+
Michael.Grundvig@... 2013-11-04, 03:46
+
Sriram Ramachandrasekaran... 2013-11-04, 04:11
+
Michael.Grundvig@... 2013-11-04, 05:43
+
Sriram Ramachandrasekaran... 2013-11-04, 05:52
+
lars hofhansl 2013-11-04, 05:26
+
Michael.Grundvig@... 2013-11-04, 05:36
+
lars hofhansl 2013-11-04, 18:02
+
Michael.Grundvig@... 2013-11-04, 18:32
+
Sriram Ramachandrasekaran... 2013-11-04, 05:43
+
Michael.Grundvig@... 2013-11-04, 05:51
+
Sriram Ramachandrasekaran... 2013-11-04, 05:59
Copy link to this message
-
Re: HBase Client Performance Bottleneck in a Single Virtual Machine
He uses HConnection.getTable()   which in turn uses the Htable constructor
HTable(*final* *byte*[] tableName, *final* HConnection connection,
*final*ExecutorService pool)

So no worry. on HTable#close() the connection wont get closed   :)

-Anoop-
On Mon, Nov 4, 2013 at 11:29 AM, Sriram Ramachandrasekaran <
[EMAIL PROTECTED]> wrote:

> HTable is the implementation of HTableInterface. I was looking at the code
> and it *indeed* closes the underlying resources on close() unless you
> create it with the ExecutorService and HConnection Option that lars
> suggested. Please do take a look at HTable constructors - that might help.
>
> P.S: I verified this on 0.94.6 code base. Hope things haven't changed b/w
> this and your version (0.94.12)
>
>
>
> On Mon, Nov 4, 2013 at 11:21 AM, <[EMAIL PROTECTED]> wrote:
>
> > Our current usage is how I would do this in a typical database app with
> > table acting like a statement. It looks like this:
> >
> > Connection connection = null;
> > HTableInterface table = null;
> > try {
> >         connection = pool.acquire();
> >         table = connection.getTable(tableName);
> >         // Do work
> > } finally {
> >         table.close();
> >         pool.release(connection);
> > }
> >
> > Is this incorrect? The API docs says close " Releases any resources held
> > or pending changes in internal buffers." I didn't interpret that as
> having
> > it close the underlying connection. Thanks!
> >
> > -Mike
> >
> > -----Original Message-----
> > From: Sriram Ramachandrasekaran [mailto:[EMAIL PROTECTED]]
> > Sent: Sunday, November 03, 2013 11:43 PM
> > To: [EMAIL PROTECTED]
> > Cc: [EMAIL PROTECTED]
> > Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> > Machine
> >
> > Hey Michael - Per API documentation, closing the HTable Instance would
> > close the underlying resources too. Hope you are aware of it.
> >
> >
> > On Mon, Nov 4, 2013 at 11:06 AM, <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi Lars, at application startup the pool is created with X number of
> > > connections using the first method you indicated:
> > > HConnectionManager.createConnection(conf). We store each connection in
> > > the pool automatically and serve it up to threads as they request it.
> > > When a thread is done using the connection, they return it back to the
> > > pool. The connections are not be created and closed per thread, but
> > > only once for the entire application. We are using the
> > > GenericObjectPool from Apache Commons Pooling as the foundation of our
> > > connection pooling approach. Our entire pool implementation really
> > > consists of just a couple overridden methods to specify how to create
> > > a new connection and close it. The GenericObjectPool class does all the
> > rest. See here for details:
> > > http://commons.apache.org/proper/commons-pool/
> > >
> > > Each thread is getting a HTableInstance as needed and then closing it
> > > when done. The only thing we are not doing is using the
> > > createConnection method that takes in an ExecutorService as that
> > > wouldn't work in our model. Our app is like a web application - the
> > > thread pool is managed outside the scope of our application code so we
> > > can't assume the service is available at connection creation time.
> > Thanks!
> > >
> > > -Mike
> > >
> > >
> > > -----Original Message-----
> > > From: lars hofhansl [mailto:[EMAIL PROTECTED]]
> > > Sent: Sunday, November 03, 2013 11:27 PM
> > > To: [EMAIL PROTECTED]
> > > Subject: Re: HBase Client Performance Bottleneck in a Single Virtual
> > > Machine
> > >
> > > Hi Micheal,
> > >
> > > can you try to create a single HConnection in your client:
> > > HConnectionManager.createConnection(Configuration conf) or
> > > HConnectionManager.createConnection(Configuration conf,
> > > ExecutorService
> > > pool)
> > >
> > > Then use HConnection.getTable(...) each time you need to do an
> operation.
> > >
> > > I.e.
> > > Configuration conf = ...;
+
Anoop John 2013-11-04, 05:58
+
Vladimir Rodionov 2013-11-04, 18:50
+
Michael.Grundvig@... 2013-11-04, 19:00
+
lars hofhansl 2013-11-05, 01:16
+
lars hofhansl 2013-11-05, 01:55
+
Vladimir Rodionov 2013-11-05, 02:41
+
Michael.Grundvig@... 2013-11-05, 14:43
+
lars hofhansl 2013-11-05, 15:46
+
Michael.Grundvig@... 2013-11-05, 16:35
+
lars hofhansl 2013-11-05, 16:45
+
lars hofhansl 2013-11-06, 19:09
+
Vladimir Rodionov 2013-11-06, 19:43
+
Vladimir Rodionov 2013-11-06, 19:56
+
Stack 2013-11-05, 15:43
+
Stack 2013-11-04, 20:11
+
Vladimir Rodionov 2013-11-04, 19:08