Raúl Gutiérrez Segalés
Raúl Gutiérrez Segalés
Has anyone done any load test on three node zookeeper cluster? In general,
I am trying to see how much requests per second, it can handle?
Any past experiences will help a lot.
I am using curator api to read from zookeeper node. And the configuration
of the machine is -
4vCPU, 12GB RAM, 60GB HD
On 25 March 2014 20:08, Check Peck <[EMAIL PROTECTED]> wrote:
I find https://github.com/phunt/zk-smoketest handy to calibrate settings on
a new cluster (specially the globalOutstandingLimit param, among others).
Thanks Raul. That's useful stuff. Does it also tell us how much throughput
it can handle?
In my use case, we have around 600 application machines and we make call to
three node zookeeper cluster every one minute from these machines. So I am
trying to understand how much load it can handle.
On Tue, Mar 25, 2014 at 8:20 PM, Raúl Gutiérrez Segalés <[EMAIL PROTECTED]
On 25 March 2014 20:41, Check Peck <[EMAIL PROTECTED]> wrote:
The latencies script will give you a good estimate of the read/write rate
your cluster can hold.
I am assuming 1 ZK session per application machine? If so, 600 sessions
shouldn't be a problem
for a 3 servers cluster. What ZK version are you planning on running?
On Tue, Mar 25, 2014 at 9:00 PM, Raúl Gutiérrez Segalés <[EMAIL PROTECTED]
Yes one ZK session per application. Zookeeper version I am running
currently is 3.4.5 in production.
Thanks that is also quite useful.
For my above use case, I just need to get the children of a particular
znode.. And that's what will be happening from all the 600 application
From the wiki you provided, it looks it should be fine right?
On Tue, Mar 25, 2014 at 10:13 PM, Rakesh R <[EMAIL PROTECTED]> wrote:
Thanks guys.. I am going to do load test in couple of days using the
exiting Python framework. And I will share the results once I am done.
On Tue, Mar 25, 2014 at 11:19 PM, Rakesh R <[EMAIL PROTECTED]> wrote:
I tried following the steps mentioned on this github link -
https://github.com/phunt/zk-smoketest for zk-latencies.py.
I have 3 node ZK cluster- machineA, machineB, machineC
Now I copied the Python program zk-latencies.py and zkclient.py in another
machineD in which no zookeeper is running. It's a plain ubuntu box.
Now when I ran the program like this -
david@machineD:~/zooksmoketest$ ./zk-latencies.py --servers
Traceback (most recent call last):
File "./zk-latencies.py", line 22, in <module>
File "/home/david/zooksmoketest/zkclient.py", line 17, in <module>
import zookeeper, time, threading
*ImportError: No module named zookeeper*
I got an error as shown above. Any thoughts what wrong I am doing here? I
don't have that much experience with Python. Any suggestions will be of
@Raul: Any thoughts?
On Sun, Apr 6, 2014 at 6:46 PM, Check Peck <[EMAIL PROTECTED]> wrote:
You need to install zkpython. The steps are described in the README
file. Let us know if you have trouble installing zkpython.
On Sun, Apr 6, 2014 at 9:21 PM, Check Peck <[EMAIL PROTECTED]> wrote:
Thanks. I tried going through that github link but wasn't able to
understand what I am supposed to do clearly. May be I am not that much
familiar with Python so that's why I am having hard time figuring that out?
On Sun, Apr 6, 2014 at 10:06 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>wrote:
Where are you getting stuck?
On Mon, Apr 7, 2014 at 11:13 AM, Check Peck <[EMAIL PROTECTED]> wrote:
I am not sure from where to start to install zkpython.. I tried following
that github link but was not able to understand. If you can guide me what I
am supposed to do step by step, then it will be of great help to me. Thanks.
On Sun, Apr 6, 2014 at 10:57 PM, Pavan Sudheendra <[EMAIL PROTECTED]>wrote:
Something like this should work on ubuntu.
On Sun, Apr 6, 2014 at 11:11 PM, Check Peck <[EMAIL PROTECTED]> wrote:
Thanks a lot Michi. Appreciated your help.. I tried your suggestion on one
of my production box but when I tried "fifth" step which is "ant". I got an
error as shown below. Any thoughts what is wrong? I guess firewall is
blocking. Is there any other way around for this? I manually copied
"zookeeper-3.4.6.tar.gz" into that machine using "scp" as well.
[get] Error getting
java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
Total time: 1 minute 3 seconds
On Mon, Apr 7, 2014 at 3:08 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>wrote:
You probably need to set the proxy.
On Mon, Apr 7, 2014 at 8:40 PM, Check Peck <[EMAIL PROTECTED]> wrote:
I manually copied the jar file into the directory it was trying to copy.
Now if I do "ant", then I see like this -
[ivy:retrieve] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/::
[ivy:retrieve] :: loading settings :: file =
[ivy:retrieve] :: resolving dependencies ::
[ivy:retrieve] confs: [default]
And then after this, it doesn't move further, I waited for a long time and
nothing came up.. And afterwards I did ctrl + c to break.
Any idea what might be wrong?
On Mon, Apr 7, 2014 at 8:42 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>wrote:
Does this work?
On Mon, Apr 7, 2014 at 9:10 PM, Check Peck <[EMAIL PROTECTED]> wrote:
Thanks a lot Michi. Yes that works like a charm. Appreciated your help.
Now coming back to my original question as I am interested in doing load
testing on my 3 node ZK cluster. So I ran "zk-latencies.py" script to do
the load test and this is what I got on my screen -
LD_LIBRARY_PATH=lib.linux-x86_64-2.6 ./zk-latencies.py --servers
"host1:2181,host2:2181,host3:2181" --znode_count=100 --znode_size=100
Connecting to host1:2181
Connected in 547 ms, handle is 0
Connecting to host2:2181
Connected in 3 ms, handle is 1
Connecting to host3:2181
Connected in 8 ms, handle is 2
Testing latencies on server host1:2181 using syncronous calls
created 100 permanent znodes in 117 ms (1.178288 ms/op
set 100 znodes in 112 ms (1.129220 ms/op
get 100 znodes in 39 ms (0.390978 ms/op
deleted 100 permanent znodes in 107 ms (1.071839 ms/op
created 100 ephemeral znodes in 111 ms (1.118169 ms/op
watched 100 znodes in 35 ms (0.354149 ms/op
deleted 100 ephemeral znodes in 110 ms (1.102810 ms/op
notif 100 watches in 1000 ms (10.007010 ms/op
Testing latencies on server host2:2181 using syncronous calls
created 100 permanent znodes in 108 ms (1.088130 ms/op
set 100 znodes in 104 ms (1.049511 ms/op
get 100 znodes in 27 ms (0.277209 ms/op
deleted 100 permanent znodes in 133 ms (1.330690 ms/op
created 100 ephemeral znodes in 102 ms (1.021519 ms/op
watched 100 znodes in 32 ms (0.327361 ms/op
deleted 100 ephemeral znodes in 110 ms (1.109312 ms/op
notif 100 watches in 1001 ms (10.010960 ms/op
Testing latencies on server host3:2181 using syncronous calls
created 100 permanent znodes in 87 ms (0.874031 ms/op
set 100 znodes in 85 ms (0.854771 ms/op
get 100 znodes in 33 ms (0.338831 ms/op
deleted 100 permanent znodes in 81 ms (0.818131 ms/op
created 100 ephemeral znodes in 81 ms (0.814011 ms/op
watched 100 znodes in 27 ms (0.279410 ms/op
deleted 100 ephemeral znodes in 99 ms (0.997062 ms/op
notif 100 watches in 1001 ms (10.011010 ms/op
Latency test complete
So my question is what does this line mean at the end? Does it say, it can
do a get request call at a rate of 2557 requests per second?
get 100 znodes in 39 ms (0.390978 ms/op
Also I was interested to see how much load a 3 node ZK cluster can handle.
In my case, most of the call will be "get" (get the data from only one
znode) on the ZK cluster So with the above test I should be able to get
some idea right?
On Mon, Apr 7, 2014 at 11:40 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>wrote:
How many ZK clients do you expect to have? I would run the same number
of zk-latency.py instances with higher number of operations to get a
more accurate picture. The performance also depends on whether you use
asynchronous reads or synchronous reads.
Actual scenario will be -
Test :- 3 node ZK cluster, 600 sessions each will do one read
operation(getChildren call) every one minute.
At what configuration I should try it out.
On Tue, Apr 8, 2014 at 6:25 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>wrote:
Ok maybe you should try exactly that instead of using zk-latency.py.
Create 600 sessions and do a get once every minute. ZooKeeper should
be able to handle that.
On Tue, Apr 8, 2014 at 7:05 PM, Check Peck <[EMAIL PROTECTED]> wrote:
How would I do this then? Is there any existing frameworks which I can use
or I need to write my own code for this?
On Tue, Apr 8, 2014 at 8:54 PM, Michi Mutsuzaki <[EMAIL PROTECTED]>wrote: