Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)


Copy link to this message
-
Re: HBase 0.94.15: writes stalls periodically even under moderate steady load (AWS EC2)
Vladimir Rodionov 2014-01-17, 18:33
I just started ycsb on our real cluster:

 ./bin/ycsb load hbase -P workloads/load20m -p columnfamily=family -s
-threads 100 -target 15000

This is beginning of a ycsb log on real grid run (looks even worse than on
AWS):
 10 sec: 150231 operations; 14984.14 current ops/sec; [INSERT
AverageLatency(us)=448.53]
 20 sec: 300299 operations; 14999.3 current ops/sec; [INSERT
AverageLatency(us)=16.21]
 30 sec: 450345 operations; 14998.6 current ops/sec; [INSERT
AverageLatency(us)=21.65]
 40 sec: 468100 operations; 1775.14 current ops/sec; [INSERT
AverageLatency(us)=12.99]
 50 sec: 750441 operations; 28225.63 current ops/sec; [INSERT
AverageLatency(us)=3747.47]
 60 sec: 900464 operations; 14997.8 current ops/sec; [INSERT
AverageLatency(us)=11.4]
 70 sec: 937443 operations; 3696.79 current ops/sec; [INSERT
AverageLatency(us)=183.94]
 80 sec: 1200550 operations; 26305.44 current ops/sec; [INSERT
AverageLatency(us)=3185.25]
 90 sec: 1350585 operations; 14999 current ops/sec; [INSERT
AverageLatency(us)=12.8]
 100 sec: 1408345 operations; 5773.69 current ops/sec; [INSERT
AverageLatency(us)=286.38]
 110 sec: 1613792 operations; 20536.49 current ops/sec; [INSERT
AverageLatency(us)=4152.39]
 120 sec: 1800751 operations; 18690.29 current ops/sec; [INSERT
AverageLatency(us)=1481.83]
 130 sec: 1874264 operations; 7349.83 current ops/sec; [INSERT
AverageLatency(us)=46.15]
 140 sec: 2100841 operations; 22650.9 current ops/sec; [INSERT
AverageLatency(us)=3326.26]
 150 sec: 2250867 operations; 14998.1 current ops/sec; [INSERT
AverageLatency(us)=12.78]
 160 sec: 2342700 operations; 9180.55 current ops/sec; [INSERT
AverageLatency(us)=64.83]
 170 sec: 2550953 operations; 20819.05 current ops/sec; [INSERT
AverageLatency(us)=3858.96]
 180 sec: 2701021 operations; 15002.3 current ops/sec; [INSERT
AverageLatency(us)=12.92]
 190 sec: 2810780 operations; 10971.51 current ops/sec; [INSERT
AverageLatency(us)=39.69]
 200 sec: 3001125 operations; 19028.79 current ops/sec; [INSERT
AverageLatency(us)=3625.68]
 210 sec: 3151151 operations; 14998.1 current ops/sec; [INSERT
AverageLatency(us)=12.56]
 220 sec: 3277541 operations; 12636.47 current ops/sec; [INSERT
AverageLatency(us)=23.76]
 230 sec: 3451248 operations; 17365.49 current ops/sec; [INSERT
AverageLatency(us)=4603.66]
 240 sec: 3601277 operations; 14998.4 current ops/sec; [INSERT
AverageLatency(us)=12.44]
 250 sec: 3745500 operations; 14193.78 current ops/sec; [INSERT
AverageLatency(us)=12.02]
 260 sec: 3745500 operations; 0 current ops/sec;
 270 sec: 4053803 operations; 30817.97 current ops/sec; [INSERT
AverageLatency(us)=3932.86]
 280 sec: 4203853 operations; 15000.5 current ops/sec; [INSERT
AverageLatency(us)=12.68]
 290 sec: 4233326 operations; 2946.42 current ops/sec; [INSERT
AverageLatency(us)=3528]
 300 sec: 4503944 operations; 27053.68 current ops/sec; [INSERT
AverageLatency(us)=3233.98]
 310 sec: 4653964 operations; 14999 current ops/sec; [INSERT
AverageLatency(us)=13.58]
 320 sec: 4692140 operations; 3802.77 current ops/sec; [INSERT
AverageLatency(us)=3139.73]
 330 sec: 4954605 operations; 26238.63 current ops/sec; [INSERT
AverageLatency(us)=2750.06]
 340 sec: 5104656 operations; 14999.1 current ops/sec; [INSERT
AverageLatency(us)=12.92]
 350 sec: 5152194 operations; 4751.9 current ops/sec; [INSERT
AverageLatency(us)=52.98]
 360 sec: 5404751 operations; 25250.65 current ops/sec; [INSERT
AverageLatency(us)=3347.95]
 370 sec: 5554789 operations; 14999.3 current ops/sec; [INSERT
AverageLatency(us)=12.68]

Unfortunately, I do not have right now time, but I will get  back to this
problem asap. I do not see any performance consistency even under moderate
load.
On Fri, Jan 17, 2014 at 10:21 AM, Andrew Purtell <[EMAIL PROTECTED]>wrote:

> I generally agree. However, the "High I/O" and "Cluster Compute" types are
> HVM single tenant on the server, and the IO stack uses SR-IOV so MMIO and
> interrupts are directly going to the VM, and 10GE network paths with no
> traffic but your own. The locally attached storage is SSD. This is pretty