Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # dev >> Something about performance of Zookeeper


Copy link to this message
-
Re: Something about performance of Zookeeper
are you using 1 gbs ethernet or 100 mbs? those numbers seem to correspond to
100 mbs. you may want to do a simple bandwidth test just to make sure.

how long are you running the test for?

also, you might check that the clients are being spread around the zk
servers. using the 4 letter works or phunts zktop.

ben

2011/3/9 lei he <[EMAIL PROTECTED]>

> Hi Flavio and Qian, I did some tests in my environment, too. And it seems
> it is not as good as it supposed to be. I used async interface(zoo_aget),
> send about 10000 requests to3  servers at the same time. But it turn out to
> be on the same level with Qian's test. Is there anything I should pay
> attentio to?
>
> thanks a lot
> He, Lei
>
> 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>
>> How many outstanding requests do you have at a time, Qian? Only one? If
>> so, you should instead have multiple outstanding. That is, you don't wait
>> for the callback to issue a new request.
>>
>> -Flavio
>>
>> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>>
>> Hi Flavio, asynchronous calls doesn't perform better, here is some results
>> we've got,
>>
>> § 1 client server,1 process per client server,connect 1 zookeeper
>> server,all reads:cpu:14%~15%,qps:3833,latency:0.000261
>> § 1 client server,1 process per client server,connect all 3 zookeeper
>> server,all reads:cpu:14%~15%,qps:3832,latency:0.000261
>> § 1 client server,10 process per client server,connect all 3 zookeeper
>> server,all reads,cpu:13%~20%,qps:14000->12000,latency:0.000469
>> *§ 1 client server,30 process per client server,connect all 3 zookeeper
>> server,all reads,cpu:15%~20%,qps:14000->10000,,latency:
>> § 2 client server,30 process per client server,connect all 3 zookeeper
>> server,all reads,cpu:15%~20%,qps:about 11000�
�atency:*
>>
>> It seems that the asynchronous calls perform even worse than the
>> synchronous calls.
>>
>>
>> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]>wrote:
>>
>>> Hi Qian, If I understand your description correctly, you are using
>>> synchronous calls. To get high throughput values, you need multiple
>>> outstanding requests, so you will need to use asynchronous calls.
>>>
>>> -Flavio
>>>
>>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>>>
>>> P.S. 1 we use zookeeper 3.3.2
>>> P.S. 2 all our testing process get data from the same znode. The size of
>>> data on the znode is less than 1K.
>>>
>>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi all:
>>>
>>>
>>> These days my friend and I did some performance tests on zookeeper. We
>>>
>>> found the performance of zookeeper is not as good as it is described in
>>> the
>>>
>>> Zookeeper Overview (
>>>
>>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) . In
>>>
>>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write Ratio
>>>
>>> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput
>>> can
>>>
>>> reach about 80000, if the requests are all reads. However, we cannot get
>>>
>>> results like that in our performance test with the synchronized
>>> interface,
>>>
>>> zkpython.
>>>
>>>
>>> Here is some of our test results:
>>>
>>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>>
>>>
>>> § 1 client server,1 process per client server,connect 1 zookeeper
>>>
>>> server,all reads:cpu:8%~9%,qps:2208,latency:0.000453s
>>>
>>> § 1 client server,1 process per client server,connect all 3 zookeeper
>>>
>>> server,all reads:cpu:8%~9%,qps:2376.241573 �
�atency:0.000421s
>>>
>>> § 1 client server,1 process per client server,connect all 3 zookeeper
>>>
>>> server,all reads,cpu:10%~20%,qps:15600,latency:0.000764s
>>>
>>> *§ 1 client server,30 process per client server,connect all 3 zookeeper
>>>
>>> server,all reads,cpu:10%~20%,qps:15200,latency:*
>>>
>>> *§ 2 client server,30 process **per client server**,connect all 3
>>>
>>> zookeeper server,all reads,cpu:10%~20%,qps:15800,latency:0.003487*
>>>
>>>
>>> qps means "query per second", that is throughput. The result shows that