Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Client Performance Bottleneck in a Single Virtual Machine


Copy link to this message
-
RE: HBase Client Performance Bottleneck in a Single Virtual Machine
One more: "hbase.ipc.client.tcpnodelay" set to true. It is worth trying as well.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: lars hofhansl [[EMAIL PROTECTED]]
Sent: Monday, November 04, 2013 5:55 PM
To: [EMAIL PROTECTED]; lars hofhansl
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Here're one more thing to try. By default each HConnection will use a single TCP connection to multiplex traffic to each region server.

Try setting hbase.client.ipc.pool.size on the client to something > 1.

-- Lars

________________________________
 From: lars hofhansl <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Monday, November 4, 2013 5:16 PM
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine
No. This is terrible.
If you can, please send a jstack and do some profiling. Is there an easy way to reproduce this with just a single RegionServer?
If so, I'd offer to do some profiling.

Thanks.
-- Lars

________________________________

From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, November 4, 2013 11:00 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine
Not yet, this is just a load test client. It literally does nothing but create threads to talk to HBase and run 4 different calls. Nothing else is done in the app at all.

To eliminate even more of our code from the loop, we just tried removing our connection pool entirely and just using a single connection per thread - no improvement. Then we tried creating the HTableInterface (all calls are against the same table) at the time of connection creation. The means thread to connection to table interface were all at 1 to 1 and not being passed around. No performance improvement.

Long story short, running a single thread it's fast. Start multithreading, it starts slowing down. CPU usage, memory usage, etc. are all negligible. The performance isn't terrible - it's probably good enough for the vast majority of users, but it's not good enough for our app. With one thread, it might take 5 milliseconds. With 10 threads all spinning more quickly (40 milliseconds delay), the call time increases to 15-30 milliseconds. The problem is that at our throughput rates, that's a serious concern.

We are going to fire up a profiler next to see what we can find.

-Mike

-----Original Message-----
From: Vladimir Rodionov [mailto:[EMAIL PROTECTED]]
Sent: Monday, November 04, 2013 12:50 PM
To: [EMAIL PROTECTED]
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

Michael, have you tried jstack on your client application?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: [EMAIL PROTECTED] [[EMAIL PROTECTED]]
Sent: Sunday, November 03, 2013 7:46 PM
To: [EMAIL PROTECTED]
Subject: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi all; I posted this as a question on StackOverflow as well but realized I should have gone straight ot the horses-mouth with my question. Sorry for the double post!

We are running a series of HBase tests to see if we can migrate one of our existing datasets from a RDBMS to HBase. We are running 15 nodes with 5 zookeepers and HBase 0.94.12 for this test.

We have a single table with three column families and a key that is distributing very well across the cluster. All of our queries are running a direct look-up; no searching or scanning. Since the HTablePool is now frowned upon, we are using the Apache commons pool and a simple connection factory to create a pool of connections and use them in our threads. Each thread creates an HTableInstance as needed and closes it when done. There are no leaks we can identify.

If we run a single thread and just do lots of random calls sequentially, the performance is quite good. Everything works great until we start trying to scale the performance. As we add more threads and try and get more work done in a single VM, we start seeing performance degrade quickly. The client code is simply attempting to run either one of several gets or a single put at a given frequency. It then waits until the next time to run, we use this to simulate the workload from external clients. With a single thread, we will see call times in the 2-3 milliseconds which is acceptable.

As we add more threads, this call time starts increasing quickly. What gets strange is if we add more VMs, the times hold steady across them all so clearly it's a bottleneck in the running instance and not the cluster. We can get a huge amount of processing happening across the cluster very easily - it just has to use a lot of VMs on the client side to do it. We know the contention isn't in the connection pool as we see the problem even when we have more connections than threads. Unfortunately, the times are spiraling out of control very quickly. We need it to support at least 128 threads in practice, but most important I want to support 500 updates/sec and 250 gets/sec. In theory, this should be a piece of cake for the cluster as we can do FAR more work than that with a few VMs, but we don't even get close to this with a single VM.

So my question: how do people building high-performance apps with HBase get around this? What approach are others using for connection pooling in a multi-threaded environment? There seems to be a surprisingly little amount of info about this on the web considering the popularity. Is there some client setting we need to use that makes it perform better in a threaded environment? We are going to try to cache HTable instances next but that's a total guess. There are solutions to offloading work to other VMs but we really want
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB