Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> HBase Thrift inserts bottlenecked somewhere -- but where?


+
Dan Crosta 2013-03-01, 12:17
+
Asaf Mesika 2013-03-01, 14:13
+
Dan Crosta 2013-03-01, 14:17
+
Jean-Daniel Cryans 2013-03-01, 17:33
+
Varun Sharma 2013-03-01, 18:46
+
Varun Sharma 2013-03-01, 18:46
+
Dan Crosta 2013-03-01, 18:49
+
Ted Yu 2013-03-01, 18:52
+
Varun Sharma 2013-03-01, 19:01
+
Ted Yu 2013-03-02, 03:53
+
Dan Crosta 2013-03-02, 17:15
+
lars hofhansl 2013-03-02, 03:42
Copy link to this message
-
Re: HBase Thrift inserts bottlenecked somewhere -- but where?
On Mar 1, 2013, at 10:42 PM, lars hofhansl wrote:
> What performance profile do you expect?

That's a good question. Our configuration is actually already exceeding our minimum and desired performance thresholds, so I'm not too worried about it. My concern is more that I develop an understanding of where the bottlenecks are (e.g. it doesn't appear to be disk, CPU, or network bound at the moment), and develop an intuition for working with HBase in case we are ever under the gun.
> Where does it top out (i.e. how many ops/sec)?

We're doing about 20,000 writes per second sustained across 4 tables and 6 CFs. Does this sound ballpark right for 6x EC2 m1.xlarges?
> Also note that each data item is replicated to three nodes (by HDFS). So in a 6 machine cluster each machine would get 50% of the writes.
> If you are looking for performance you really need a larger cluster to amortize this replication cost across more machines.

That's only true from the HDFS perspective, right? Any given region is "owned" by 1 of the 6 regionservers at any given time, and writes are buffered to memory before being persisted to HDFS, right?

In any event, there doesn't seem to be any disk contention to speak of -- we average around 10% disk utilization at this level of load (each machine has 4 spindles of local storage, we are not using EBS).

One setting no one has mentioned yet is the DataNode handler count (dfs.datanode.handler.count) -- which is currently set to its default of 3. Should we experiment with raising that?
> The other issue to watch out for is whether your keys are generated such that a single regionserver is hot spotted (you can look at the operation count on the master page).

All of our keys are hashes or UUIDs, so the key distribution is very smooth, and this is confirmed by the "Region Servers" table on the master node's web UI.
Thanks,
- Dan
+
lars hofhansl 2013-03-02, 17:38
+
Dan Crosta 2013-03-02, 18:47
+
Asaf Mesika 2013-03-02, 19:56
+
Ted Yu 2013-03-02, 20:02
+
lars hofhansl 2013-03-02, 20:50
+
lars hofhansl 2013-03-02, 20:50
+
Dan Crosta 2013-03-02, 22:29
+
Varun Sharma 2013-03-03, 11:08
+
Dan Crosta 2013-03-03, 13:53
+
lars hofhansl 2013-03-02, 20:56
+
Andrew Purtell 2013-03-05, 07:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB