Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase Client Performance Bottleneck in a Single Virtual Machine


+
Michael.Grundvig@... 2013-11-04, 03:46
+
Sriram Ramachandrasekaran... 2013-11-04, 04:11
+
Michael.Grundvig@... 2013-11-04, 05:43
+
Sriram Ramachandrasekaran... 2013-11-04, 05:52
+
lars hofhansl 2013-11-04, 05:26
+
Michael.Grundvig@... 2013-11-04, 05:36
+
lars hofhansl 2013-11-04, 18:02
+
Michael.Grundvig@... 2013-11-04, 18:32
+
Sriram Ramachandrasekaran... 2013-11-04, 05:43
+
Michael.Grundvig@... 2013-11-04, 05:51
+
Sriram Ramachandrasekaran... 2013-11-04, 05:59
+
Anoop John 2013-11-04, 06:06
+
Anoop John 2013-11-04, 05:58
+
Vladimir Rodionov 2013-11-04, 18:50
+
Michael.Grundvig@... 2013-11-04, 19:00
+
lars hofhansl 2013-11-05, 01:16
+
lars hofhansl 2013-11-05, 01:55
+
Vladimir Rodionov 2013-11-05, 02:41
+
Michael.Grundvig@... 2013-11-05, 14:43
+
lars hofhansl 2013-11-05, 15:46
+
Michael.Grundvig@... 2013-11-05, 16:35
+
lars hofhansl 2013-11-05, 16:45
Copy link to this message
-
Re: HBase Client Performance Bottleneck in a Single Virtual Machine
If you get the chance, please send a jstack of the client when it is slow, or some profile data if you have, or just some code reproducing the problem.
We should track this down.

-- Lars

________________________________
 From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Tuesday, November 5, 2013 8:35 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine
 

I'm finding this configuration process very confusing. This is what we are doing in Java code:

        Configuration configuration = HBaseConfiguration.create();
        configuration.set("hbase.zookeeper.quorum", props.getProperty(HBASE_NODES));
        configuration.set("zookeeper.znode.parent", props.getProperty(HBASE_PARAMS));
        configuration.setBoolean(" hbase.ipc.client.tcpnodelay", true);
        configuration.setInt("hbase.client.ipc.pool.size", 10);

Adding those last two lines has made no improvement. Likewise, updating hbase-site.xml on the cluster to include these settings has made no improvement. The problem is that I have zero confidence it's actually taking the settings in the first place.

-Mike

-----Original Message-----
From: lars hofhansl [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, November 05, 2013 9:47 AM
To: [EMAIL PROTECTED]
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi Mike,

hbase-site.xml has both client and server side settings.

In your client, where do you set the ZK quorum of the cluster? At that same spot you'd set these options.

Assuming you're running the clients with classes from a normal HBase install, you would add these setting to the hbase-site.xml there.

-- Lars

________________________________
From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Tuesday, November 5, 2013 6:43 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine
Thanks for the input, we'll do some more testing on this today. Where do these settings get made? In the configuration used to create the pool or somewhere else? It appears hbase-site.xml "on the client" but we have nothing like that so I think I'm misunderstanding something. Thanks!

-Mike

-----Original Message-----
From: Vladimir Rodionov [mailto:[EMAIL PROTECTED]]
Sent: Monday, November 04, 2013 8:41 PM
To: [EMAIL PROTECTED]; lars hofhansl
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

One more: "hbase.ipc.client.tcpnodelay" set to true. It is worth trying as well.

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: lars hofhansl [[EMAIL PROTECTED]]
Sent: Monday, November 04, 2013 5:55 PM
To: [EMAIL PROTECTED]; lars hofhansl
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine

Here're one more thing to try. By default each HConnection will use a single TCP connection to multiplex traffic to each region server.

Try setting hbase.client.ipc.pool.size on the client to something > 1.

-- Lars

________________________________
From: lars hofhansl <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
Sent: Monday, November 4, 2013 5:16 PM
Subject: Re: HBase Client Performance Bottleneck in a Single Virtual Machine
No. This is terrible.
If you can, please send a jstack and do some profiling. Is there an easy way to reproduce this with just a single RegionServer?
If so, I'd offer to do some profiling.

Thanks.
-- Lars

________________________________

From: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, November 4, 2013 11:00 AM
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine
Not yet, this is just a load test client. It literally does nothing but create threads to talk to HBase and run 4 different calls. Nothing else is done in the app at all.

To eliminate even more of our code from the loop, we just tried removing our connection pool entirely and just using a single connection per thread - no improvement. Then we tried creating the HTableInterface (all calls are against the same table) at the time of connection creation. The means thread to connection to table interface were all at 1 to 1 and not being passed around. No performance improvement.

Long story short, running a single thread it's fast. Start multithreading, it starts slowing down. CPU usage, memory usage, etc. are all negligible. The performance isn't terrible - it's probably good enough for the vast majority of users, but it's not good enough for our app. With one thread, it might take 5 milliseconds. With 10 threads all spinning more quickly (40 milliseconds delay), the call time increases to 15-30 milliseconds. The problem is that at our throughput rates, that's a serious concern.

We are going to fire up a profiler next to see what we can find.

-Mike

From: Vladimir Rodionov [mailto:[EMAIL PROTECTED]]
Sent: Monday, November 04, 2013 12:50 PM
To: [EMAIL PROTECTED]
Subject: RE: HBase Client Performance Bottleneck in a Single Virtual Machine

Michael, have you tried jstack on your client application?

Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: [EMAIL PROTECTED]

________________________________________
From: [EMAIL PROTECTED] [[EMAIL PROTECTED]]
Sent: Sunday, November 03, 2013 7:46 PM
To: [EMAIL PROTECTED]
Subject: HBase Client Performance Bottleneck in a Single Virtual Machine

Hi all; I posted this as a question on StackOverflow as well but realized I should have gone straight ot the horses-mouth with my question. Sorry for the double post!

We are running a series of HBase tests to see if we can migrate one of our existing datasets from a RDBMS to HBase. We are running 15 nodes with
+
Vladimir Rodionov 2013-11-06, 19:43
+
Vladimir Rodionov 2013-11-06, 19:56
+
Stack 2013-11-05, 15:43
+
Stack 2013-11-04, 20:11
+
Vladimir Rodionov 2013-11-04, 19:08
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB