Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)


Copy link to this message
-
Re: HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)
I just started ycsb on our real cluster:

 ./bin/ycsb load hbase -P workloads/load20m -p columnfamily=family -s
-threads 100 -target 15000

This is beginning of a ycsb log on real grid run (looks even worse than on
AWS):
 10 sec: 150231 operations; 14984.14 current ops/sec; [INSERT
AverageLatency(us)=448.53]
 20 sec: 300299 operations; 14999.3 current ops/sec; [INSERT
AverageLatency(us)=16.21]
 30 sec: 450345 operations; 14998.6 current ops/sec; [INSERT
AverageLatency(us)=21.65]
 40 sec: 468100 operations; 1775.14 current ops/sec; [INSERT
AverageLatency(us)=12.99]
 50 sec: 750441 operations; 28225.63 current ops/sec; [INSERT
AverageLatency(us)=3747.47]
 60 sec: 900464 operations; 14997.8 current ops/sec; [INSERT
AverageLatency(us)=11.4]
 70 sec: 937443 operations; 3696.79 current ops/sec; [INSERT
AverageLatency(us)=183.94]
 80 sec: 1200550 operations; 26305.44 current ops/sec; [INSERT
AverageLatency(us)=3185.25]
 90 sec: 1350585 operations; 14999 current ops/sec; [INSERT
AverageLatency(us)=12.8]
 100 sec: 1408345 operations; 5773.69 current ops/sec; [INSERT
AverageLatency(us)=286.38]
 110 sec: 1613792 operations; 20536.49 current ops/sec; [INSERT
AverageLatency(us)=4152.39]
 120 sec: 1800751 operations; 18690.29 current ops/sec; [INSERT
AverageLatency(us)=1481.83]
 130 sec: 1874264 operations; 7349.83 current ops/sec; [INSERT
AverageLatency(us)=46.15]
 140 sec: 2100841 operations; 22650.9 current ops/sec; [INSERT
AverageLatency(us)=3326.26]
 150 sec: 2250867 operations; 14998.1 current ops/sec; [INSERT
AverageLatency(us)=12.78]
 160 sec: 2342700 operations; 9180.55 current ops/sec; [INSERT
AverageLatency(us)=64.83]
 170 sec: 2550953 operations; 20819.05 current ops/sec; [INSERT
AverageLatency(us)=3858.96]
 180 sec: 2701021 operations; 15002.3 current ops/sec; [INSERT
AverageLatency(us)=12.92]
 190 sec: 2810780 operations; 10971.51 current ops/sec; [INSERT
AverageLatency(us)=39.69]
 200 sec: 3001125 operations; 19028.79 current ops/sec; [INSERT
AverageLatency(us)=3625.68]
 210 sec: 3151151 operations; 14998.1 current ops/sec; [INSERT
AverageLatency(us)=12.56]
 220 sec: 3277541 operations; 12636.47 current ops/sec; [INSERT
AverageLatency(us)=23.76]
 230 sec: 3451248 operations; 17365.49 current ops/sec; [INSERT
AverageLatency(us)=4603.66]
 240 sec: 3601277 operations; 14998.4 current ops/sec; [INSERT
AverageLatency(us)=12.44]
 250 sec: 3745500 operations; 14193.78 current ops/sec; [INSERT
AverageLatency(us)=12.02]
 260 sec: 3745500 operations; 0 current ops/sec;
 270 sec: 4053803 operations; 30817.97 current ops/sec; [INSERT
AverageLatency(us)=3932.86]
 280 sec: 4203853 operations; 15000.5 current ops/sec; [INSERT
AverageLatency(us)=12.68]
 290 sec: 4233326 operations; 2946.42 current ops/sec; [INSERT
AverageLatency(us)=3528]
 300 sec: 4503944 operations; 27053.68 current ops/sec; [INSERT
AverageLatency(us)=3233.98]
 310 sec: 4653964 operations; 14999 current ops/sec; [INSERT
AverageLatency(us)=13.58]
 320 sec: 4692140 operations; 3802.77 current ops/sec; [INSERT
AverageLatency(us)=3139.73]
 330 sec: 4954605 operations; 26238.63 current ops/sec; [INSERT
AverageLatency(us)=2750.06]
 340 sec: 5104656 operations; 14999.1 current ops/sec; [INSERT
AverageLatency(us)=12.92]
 350 sec: 5152194 operations; 4751.9 current ops/sec; [INSERT
AverageLatency(us)=52.98]
 360 sec: 5404751 operations; 25250.65 current ops/sec; [INSERT
AverageLatency(us)=3347.95]
 370 sec: 5554789 operations; 14999.3 current ops/sec; [INSERT
AverageLatency(us)=12.68]

Unfortunately, I do not have right now time, but I will get  back to this
problem asap. I do not see any performance consistency even under moderate
load.
On Fri, Jan 17, 2014 at 10:21 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> I generally agree. However, the "High I/O" and "Cluster Compute" types are
> HVM single tenant on the server, and the IO stack uses SR-IOV so MMIO and
> interrupts are directly going to the VM, and 10GE network paths with no
> traffic but your own. The locally attached storage is SSD. This is pretty
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB