Hi all:

These days my friend and I did some performance tests on zookeeper. We found
the performance of zookeeper is not as good as it is described in the
Zookeeper Overview (
http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) . In the
Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write Ratio
Varies" shows that in a ensemble of 3 Zookeeper server, the throughput can
reach about 80000, if the requests are all reads. However, we cannot get
results like that in our performance test with the synchronized interface,
zkpython.

Here is some of our test results:
(3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)

§ 1 client server，1 process per client server，connect 1 zookeeper server，all
reads：cpu：8%~9%，qps：2208，latency���0.000453s
§ 1 client server，1 process per client server，connect all 3 zookeeper
server，all reads：cpu：8%~9%，qps：2376.241573 ，latency:0.000421s
§ 1 client server，1 process per client server，connect all 3 zookeeper
server，all reads，cpu：10%~20%，qps：15600，latency：0.000764s
*§ 1 client server，30 process per client server，connect all 3 zookeeper
server，all reads，cpu：10%~20%，qps：15200，latency：*
*§ 2 client server，30 process **per client server**，connect all 3 zookeeper
server，all reads，cpu：10%~20%，qps：15800，latency：0.003487*

qps means "query per second", that is throughput. The result shows that when
adding more client server, the utilization rate of CPU don't increase,  and
the throughput don't increase much. It seems that the throughput won't reach
80000, even if we add 28 more client servers to reach the number you
mentioned in the Zookeeper Overview.

Maybe I've done the tests wrong. Is there any particular thing I should pay
attention to in this case? We set the max java heap size to 12GB in our
test.

*Could you tell me the details about how you do the performance test, from
which you get the results showed in the Zookeeper Overview?*

--
With Regards!

Ye, Qian

P.S. 1 we use zookeeper 3.3.2
P.S. 2 all our testing process get data from the same znode. The size of
data on the znode is less than 1K.

On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:

> Hi all:
>
> These days my friend and I did some performance tests on zookeeper. We
> found the performance of zookeeper is not as good as it is described in the
> Zookeeper Overview (
> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) . In
> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write Ratio
> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput can
> reach about 80000, if the requests are all reads. However, we cannot get
> results like that in our performance test with the synchronized interface,
> zkpython.
>
> Here is some of our test results:
> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>
> § 1 client server，1 process per client server，connect 1 zookeeper
> server，all reads：cpu：8%~9%，qps：2208，latency：0.000453s
> § 1 client server，1 process per client server，connect all 3 zookeeper
> server，all reads：cpu：8%~9%，qps：2376.241573 ，latency:0.000421s
> § 1 client server，1 process per client server，connect all 3 zookeeper
> server，all reads，cpu：10%~20%，qps：15600，latency：0.000764s
> *§ 1 client server，30 process per client server，connect all 3 zookeeper
> server，all reads，cpu：10%~20%，qps：15200，latency：*
> *§ 2 client server，30 process **per client server**，connect all 3
> zookeeper server，all reads，cpu：10%~20%，qps：15800，latency：0.003487*
>
> qps means "query per second", that is throughput. The result shows that
> when adding more client server, the utilization rate of CPU don't increase,
> and the throughput don't increase much. It seems that the throughput won't
> reach 80000, even if we add 28 more client servers to reach the number you
> mentioned in the Zookeeper Overview.
>
> Maybe I've done the tests wrong. Is there any particular thing I should pay
> attention to in this case? We set the max java heap size to 12GB in our
> test.
>
> *Could you tell me the details about how you do the performance test, from
> which you get the results showed in the Zookeeper Overview?*
>
> --
> With Regards!
>
> Ye, Qian
>
>
--
With Regards!

Ye, Qian

Hi Qian, If I understand your description correctly, you are using  
synchronous calls. To get high throughput values, you need multiple  
outstanding requests, so you will need to use asynchronous calls.

-Flavio

On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:

> P.S. 1 we use zookeeper 3.3.2
> P.S. 2 all our testing process get data from the same znode. The  
> size of
> data on the znode is less than 1K.
>
> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>
>> Hi all:
>>
>> These days my friend and I did some performance tests on zookeeper.  
>> We
>> found the performance of zookeeper is not as good as it is  
>> described in the
>> Zookeeper Overview (
>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/
>> zookeeperOver.html) . In
>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write  
>> Ratio
>> Varies" shows that in a ensemble of 3 Zookeeper server, the  
>> throughput can
>> reach about 80000, if the requests are all reads. However, we  
>> cannot get
>> results like that in our performance test with the synchronized  
>> interface,
>> zkpython.
>>
>> Here is some of our test results:
>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>
>> § 1 client server，1 process per client server，connect 1  
>> zookeeper
>> server，all reads：cpu：8%~9%，qps：2208，latency���0.000453s
>> § 1 client server，1 process per client server，connect all 3  
>> zookeeper
>> server，all reads：cpu：8%~9%，qps：2376.241573 ，latency:
>> 0.000421s
>> § 1 client server，1 process per client server，connect all 3  
>> zookeeper
>> server，all reads，cpu：10%~20%，qps：15600，latency：
>> 0.000764s
>> *§ 1 client server，30 process per client server，connect all 3  
>> zookeeper
>> server，all reads，cpu：10%~20%，qps：15200，latency：*
>> *§ 2 client server，30 process **per client server**，connect all  
>> 3
>> zookeeper server，all reads，cpu：10%~20%，qps：
>> 15800，latency：0.003487*
>>
>> qps means "query per second", that is throughput. The result shows  
>> that
>> when adding more client server, the utilization rate of CPU don't  
>> increase,
>> and the throughput don't increase much. It seems that the  
>> throughput won't
>> reach 80000, even if we add 28 more client servers to reach the  
>> number you
>> mentioned in the Zookeeper Overview.
>>
>> Maybe I've done the tests wrong. Is there any particular thing I  
>> should pay
>> attention to in this case? We set the max java heap size to 12GB in  
>> our
>> test.
>>
>> *Could you tell me the details about how you do the performance  
>> test, from
>> which you get the results showed in the Zookeeper Overview?*
>>
>> --
>> With Regards!
>>
>> Ye, Qian
>>
>>
>
>
> --
> With Regards!
>
> Ye, Qian

flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301

Hi Flavio, asynchronous calls doesn't perform better, here is some results
we've got,

§ 1 client server，1 process per client server，connect 1 zookeeper server，all
reads：cpu：14%~15%，qps：3833，latency：0.000261
§ 1 client server，1 process per client server，connect all 3 zookeeper
server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
§ 1 client server，10 process per client server，connect all 3 zookeeper
server，all reads，cpu：13%~20%，qps：14000->12000，latency：0.000469
*§ 1 client server，30 process per client server，connect all 3 zookeeper
server，all reads，cpu：15%~20%，qps：14000->10000，,latency：
§ 2 client server，30 process per client server，connect all 3 zookeeper
server，all reads，cpu：15%~20%，qps：about 11000，latency：*

It seems that the asynchronous calls perform even worse than the synchronous
calls.
On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]> wrote:

> Hi Qian, If I understand your description correctly, you are using
> synchronous calls. To get high throughput values, you need multiple
> outstanding requests, so you will need to use asynchronous calls.
>
> -Flavio
>
> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>
> P.S. 1 we use zookeeper 3.3.2
> P.S. 2 all our testing process get data from the same znode. The size of
> data on the znode is less than 1K.
>
> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>
> Hi all:
>
>
> These days my friend and I did some performance tests on zookeeper. We
>
> found the performance of zookeeper is not as good as it is described in the
>
> Zookeeper Overview (
>
> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) . In
>
> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write Ratio
>
> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput can
>
> reach about 80000, if the requests are all reads. However, we cannot get
>
> results like that in our performance test with the synchronized interface,
>
> zkpython.
>
>
> Here is some of our test results:
>
> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>
>
> § 1 client server，1 process per client server，connect 1 zookeeper
>
> server，all reads：cpu：8%~9%，qps：2208，latency：0.000453s
>
> § 1 client server，1 process per client server，connect all 3 zookeeper
>
> server，all reads：cpu：8%~9%，qps：2376.241573 ，latency:0.000421s
>
> § 1 client server，1 process per client server，connect all 3 zookeeper
>
> server，all reads，cpu：10%~20%，qps：15600，latency：0.000764s
>
> *§ 1 client server，30 process per client server，connect all 3 zookeeper
>
> server，all reads，cpu：10%~20%，qps：15200，latency：*
>
> *§ 2 client server，30 process **per client server**，connect all 3
>
> zookeeper server，all reads，cpu：10%~20%，qps：15800，latency：0.003487*
>
>
> qps means "query per second", that is throughput. The result shows that
>
> when adding more client server, the utilization rate of CPU don't increase,
>
> and the throughput don't increase much. It seems that the throughput won't
>
> reach 80000, even if we add 28 more client servers to reach the number you
>
> mentioned in the Zookeeper Overview.
>
>
> Maybe I've done the tests wrong. Is there any particular thing I should pay
>
> attention to in this case? We set the max java heap size to 12GB in our
>
> test.
>
>
> *Could you tell me the details about how you do the performance test, from
>
> which you get the results showed in the Zookeeper Overview?*
>
>
> --
>
> With Regards!
>
>
> Ye, Qian
>
>
>
>
>
> --
> With Regards!
>
> Ye, Qian
>
>
> *flavio*
> *junqueira*
>
> research scientist
>
> [EMAIL PROTECTED]
> direct +34 93-183-8828
>
> avinguda diagonal 177, 8th floor, barcelona, 08018, es
> phone (408) 349 3300    fax (408) 349 3301
>
>
>
--
With Regards!

Ye, Qian

Dear Qian Ye,
     Thank you for doing these tests. I am interested to hear more about
this! Can you document how you did your testing with zkpython?
-Eugene

On 3/8/11 6:44 PM, Qian Ye wrote:
>
>>>     http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html)
>>>     <http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html%29>
>>>     . In
>>>     the Zookeeper Overview, the "ZooKeeper Throughput as the
>>>     Read-Write Ratio
>>>     Varies" shows that in a ensemble of 3 Zookeeper server, the
>>>     throughput can
>>>     reach about 80000, if the requests are all reads. However, we
>>>     cannot get
>>>     results like that in our performance test with the synchronized
>>>     interface,
>>>     zkpython.
>

How many outstanding requests do you have at a time, Qian? Only one?  
If so, you should instead have multiple outstanding. That is, you  
don't wait for the callback to issue a new request.

-Flavio

On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:

> Hi Flavio, asynchronous calls doesn't perform better, here is some  
> results we've got,
>
> § 1 client server，1 process per client server，connect 1  
> zookeeper server，all reads：cpu：14%~15%，qps：3833，latency：
> 0.000261
> § 1 client server，1 process per client server，connect all 3  
> zookeeper server，all reads：cpu：14%~15%，qps：3832，latency：
> 0.000261
> § 1 client server，10 process per client server，connect all 3  
> zookeeper server，all reads，cpu：13%~20%，qps：14000-
> >12000，latency：0.000469
> § 1 client server，30 process per client server，connect all 3  
> zookeeper server，all reads，cpu：15%~20%，qps：14000-
> >10000，,latency：
> § 2 client server，30 process per client server，connect all 3  
> zookeeper server，all reads，cpu：15%~20%，qps：about  
> 11000，latency：
>
> It seems that the asynchronous calls perform even worse than the  
> synchronous calls.
>
>
> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <fpj@yahoo-
> inc.com> wrote:
> Hi Qian, If I understand your description correctly, you are using  
> synchronous calls. To get high throughput values, you need multiple  
> outstanding requests, so you will need to use asynchronous calls.
>
> -Flavio
>
> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>
>> P.S. 1 we use zookeeper 3.3.2
>> P.S. 2 all our testing process get data from the same znode. The  
>> size of
>> data on the znode is less than 1K.
>>
>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]>  
>> wrote:
>>
>>> Hi all:
>>>
>>> These days my friend and I did some performance tests on  
>>> zookeeper. We
>>> found the performance of zookeeper is not as good as it is  
>>> described in the
>>> Zookeeper Overview (
>>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/
>>> zookeeperOver.html) . In
>>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-
>>> Write Ratio
>>> Varies" shows that in a ensemble of 3 Zookeeper server, the  
>>> throughput can
>>> reach about 80000, if the requests are all reads. However, we  
>>> cannot get
>>> results like that in our performance test with the synchronized  
>>> interface,
>>> zkpython.
>>>
>>> Here is some of our test results:
>>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>>
>>> § 1 client server，1 process per client server，connect 1  
>>> zookeeper
>>> server，all reads：cpu：8%~9%，qps：2208，latency���0.000453s
>>> § 1 client server，1 process per client server，connect all 3  
>>> zookeeper
>>> server，all reads：cpu：8%~9%，qps：2376.241573 ，latency:
>>> 0.000421s
>>> § 1 client server，1 process per client server，connect all 3  
>>> zookeeper
>>> server，all reads，cpu：10%~20%，qps��15600，latency：
>>> 0.000764s
>>> *§ 1 client server，30 process per client server，connect all 3  
>>> zookeeper
>>> server，all reads，cpu：10%~20%，qps��15200，latency：*
>>> *§ 2 client server，30 process **per client server**，connect  
>>> all 3
>>> zookeeper server，all reads，cpu：10%~20%，qps：
>>> 15800，latency：0.003487*
>>>
>>> qps means "query per second", that is throughput. The result shows  
>>> that
>>> when adding more client server, the utilization rate of CPU don't  
>>> increase,
>>> and the throughput don't increase much. It seems that the  
>>> throughput won't
>>> reach 80000, even if we add 28 more client servers to reach the  
>>> number you
>>> mentioned in the Zookeeper Overview.
>>>
>>> Maybe I've done the tests wrong. Is there any particular thing I  
>>> should pay
>>> attention to in this case? We set the max java heap size to 12GB  
>>> in our
>>> test.
>>>
>>> *Could you tell me the details about how you do the performance  
>>> test, from
>>> which you get the results showed in the Zookeeper Overview?*
>>>
>>> --
>>> With Regards!
>>>
>>> Ye, Qian
>>>
>>>
>>
>>
>> --
>> With Regards!
>>
>> Ye, Qian

flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301

Hi Flavio and Qian, I did some tests in my environment, too. And it seems it
is not as good as it supposed to be. I used async interface(zoo_aget), send
about 10000 requests to3  servers at the same time. But it turn out to be on
the same level with Qian's test. Is there anything I should pay attentio to?

thanks a lot
He, Lei

2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>

> How many outstanding requests do you have at a time, Qian? Only one? If so,
> you should instead have multiple outstanding. That is, you don't wait for
> the callback to issue a new request.
>
> -Flavio
>
> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>
> Hi Flavio, asynchronous calls doesn't perform better, here is some results
> we've got,
>
> § 1 client server，1 process per client server，connect 1 zookeeper
> server，all reads：cpu：14%~15%，qps：3833，latency：0.000261
> § 1 client server，1 process per client server，connect all 3 zookeeper
> server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
> § 1 client server，10 process per client server，connect all 3 zookeeper
> server，all reads，cpu：13%~20%，qps：14000->12000�
�atency：0.000469
> *§ 1 client server，30 process per client server，connect all 3 zookeeper
> server，all reads，cpu：15%~20%，qps：14000->10000�
�latency：
> § 2 client server，30 process per client server，connect all 3 zookeeper
> server，all reads，cpu：15%~20%，qps：about 11000�
�atency：*
>
> It seems that the asynchronous calls perform even worse than the
> synchronous calls.
>
>
> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]>wrote:
>
>> Hi Qian, If I understand your description correctly, you are using
>> synchronous calls. To get high throughput values, you need multiple
>> outstanding requests, so you will need to use asynchronous calls.
>>
>> -Flavio
>>
>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>>
>> P.S. 1 we use zookeeper 3.3.2
>> P.S. 2 all our testing process get data from the same znode. The size of
>> data on the znode is less than 1K.
>>
>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>>
>> Hi all:
>>
>>
>> These days my friend and I did some performance tests on zookeeper. We
>>
>> found the performance of zookeeper is not as good as it is described in
>> the
>>
>> Zookeeper Overview (
>>
>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) . In
>>
>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write Ratio
>>
>> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput can
>>
>> reach about 80000, if the requests are all reads. However, we cannot get
>>
>> results like that in our performance test with the synchronized interface,
>>
>> zkpython.
>>
>>
>> Here is some of our test results:
>>
>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>
>>
>> § 1 client server，1 process per client server，connect 1 zookeeper
>>
>> server，all reads：cpu：8%~9%，qps：2208，latency：0.000453s
>>
>> § 1 client server，1 process per client server，connect all 3 zookeeper
>>
>> server，all reads：cpu：8%~9%，qps：2376.241573 �
�atency:0.000421s
>>
>> § 1 client server，1 process per client server，connect all 3 zookeeper
>>
>> server，all reads，cpu：10%~20%，qps：15600，latency：0.000764s
>>
>> *§ 1 client server，30 process per client server，connect all 3 zookeeper
>>
>> server，all reads，cpu：10%~20%，qps：15200，latency：*
>>
>> *§ 2 client server，30 process **per client server**，connect all 3
>>
>> zookeeper server，all reads，cpu：10%~20%，qps：15800，latency：0.003487*
>>
>>
>> qps means "query per second", that is throughput. The result shows that
>>
>> when adding more client server, the utilization rate of CPU don't
>> increase,
>>
>> and the throughput don't increase much. It seems that the throughput won't
>>
>> reach 80000, even if we add 28 more client servers to reach the number you
>>
>> mentioned in the Zookeeper Overview.
>>
>>
>> Maybe I've done the tests wrong. Is there any particular thing I should
>> pay
>>
>> attention to in this case? We set the max java heap size to 12GB in our

are you using 1 gbs ethernet or 100 mbs? those numbers seem to correspond to
100 mbs. you may want to do a simple bandwidth test just to make sure.

how long are you running the test for?

also, you might check that the clients are being spread around the zk
servers. using the 4 letter works or phunts zktop.

ben

2011/3/9 lei he <[EMAIL PROTECTED]>

> Hi Flavio and Qian, I did some tests in my environment, too. And it seems
> it is not as good as it supposed to be. I used async interface(zoo_aget),
> send about 10000 requests to3  servers at the same time. But it turn out to
> be on the same level with Qian's test. Is there anything I should pay
> attentio to?
>
> thanks a lot
> He, Lei
>
> 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>
>> How many outstanding requests do you have at a time, Qian? Only one? If
>> so, you should instead have multiple outstanding. That is, you don't wait
>> for the callback to issue a new request.
>>
>> -Flavio
>>
>> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>>
>> Hi Flavio, asynchronous calls doesn't perform better, here is some results
>> we've got,
>>
>> § 1 client server，1 process per client server，connect 1 zookeeper
>> server，all reads：cpu：14%~15%，qps：3833，latency：0.000261
>> § 1 client server，1 process per client server，connect all 3 zookeeper
>> server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
>> § 1 client server，10 process per client server，connect all 3 zookeeper
>> server，all reads，cpu：13%~20%，qps：14000->12000，latency：0.000469
>> *§ 1 client server，30 process per client server，connect all 3 zookeeper
>> server，all reads，cpu：15%~20%，qps：14000->10000，,latency：
>> § 2 client server，30 process per client server，connect all 3 zookeeper
>> server，all reads，cpu：15%~20%，qps：about 11000�
�atency：*
>>
>> It seems that the asynchronous calls perform even worse than the
>> synchronous calls.
>>
>>
>> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]>wrote:
>>
>>> Hi Qian, If I understand your description correctly, you are using
>>> synchronous calls. To get high throughput values, you need multiple
>>> outstanding requests, so you will need to use asynchronous calls.
>>>
>>> -Flavio
>>>
>>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>>>
>>> P.S. 1 we use zookeeper 3.3.2
>>> P.S. 2 all our testing process get data from the same znode. The size of
>>> data on the znode is less than 1K.
>>>
>>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi all:
>>>
>>>
>>> These days my friend and I did some performance tests on zookeeper. We
>>>
>>> found the performance of zookeeper is not as good as it is described in
>>> the
>>>
>>> Zookeeper Overview (
>>>
>>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) . In
>>>
>>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write Ratio
>>>
>>> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput
>>> can
>>>
>>> reach about 80000, if the requests are all reads. However, we cannot get
>>>
>>> results like that in our performance test with the synchronized
>>> interface,
>>>
>>> zkpython.
>>>
>>>
>>> Here is some of our test results:
>>>
>>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>>
>>>
>>> § 1 client server，1 process per client server，connect 1 zookeeper
>>>
>>> server，all reads：cpu：8%~9%，qps：2208，latency：0.000453s
>>>
>>> § 1 client server，1 process per client server，connect all 3 zookeeper
>>>
>>> server，all reads：cpu：8%~9%，qps：2376.241573 �
�atency:0.000421s
>>>
>>> § 1 client server，1 process per client server，connect all 3 zookeeper
>>>
>>> server，all reads，cpu：10%~20%，qps：15600，latency：0.000764s
>>>
>>> *§ 1 client server，30 process per client server，connect all 3 zookeeper
>>>
>>> server，all reads，cpu：10%~20%，qps：15200，latency：*
>>>
>>> *§ 2 client server，30 process **per client server**，connect all 3
>>>
>>> zookeeper server，all reads，cpu：10%~20%，qps：15800，latency：0.003487*
>>>
>>>
>>> qps means "query per second", that is throughput. The result shows that

thanks for reply. I use scp to check my ethernet again, the transmission
rate is about 60MB/s. I send about 100000 requests in each process(60
processes in 2 clients), and it costs about 550 seconds to finish all the
requests. And I am sure that all the processes had been spread around all
the servers.

Thanks
Lei, He

在 2011年3月9日 上午9:22，Benjamin Reed <[EMAIL PROTECTED]>写道：

> are you using 1 gbs ethernet or 100 mbs? those numbers seem to correspond
> to
> 100 mbs. you may want to do a simple bandwidth test just to make sure.
>
> how long are you running the test for?
>
> also, you might check that the clients are being spread around the zk
> servers. using the 4 letter works or phunts zktop.
>
> ben
>
> 2011/3/9 lei he <[EMAIL PROTECTED]>
>
> > Hi Flavio and Qian, I did some tests in my environment, too. And it seems
> > it is not as good as it supposed to be. I used async interface(zoo_aget),
> > send about 10000 requests to3  servers at the same time. But it turn out
> to
> > be on the same level with Qian's test. Is there anything I should pay
> > attentio to?
> >
> > thanks a lot
> > He, Lei
> >
> > 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
> >
> >> How many outstanding requests do you have at a time, Qian? Only one? If
> >> so, you should instead have multiple outstanding. That is, you don't
> wait
> >> for the callback to issue a new request.
> >>
> >> -Flavio
> >>
> >> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
> >>
> >> Hi Flavio, asynchronous calls doesn't perform better, here is some
> results
> >> we've got,
> >>
> >> § 1 client server，1 process per client server，connect 1 zookeeper
> >> server，all reads：cpu：14%~15%，qps：3833，latency：0.000261
> >> § 1 client server，1 process per client server，connect all 3 zookeeper
> >> server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
> >> § 1 client server，10 process per client server，connect all 3 zookeeper
> >> server，all reads，cpu：13%~20%，qps：14000->12000，latency：0.000469
> >> *§ 1 client server，30 process per client server，connect all 3 zookeeper
> >> server，all reads，cpu：15%~20%，qps：14000->10000，,latency：
> >> § 2 client server，30 process per client server，connect all 3 zookeeper
> >> server，all reads，cpu：15%~20%，qps：about 11000，latency：*
> >>
> >> It seems that the asynchronous calls perform even worse than the
> >> synchronous calls.
> >>
> >>
> >> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]
> >wrote:
> >>
> >>> Hi Qian, If I understand your description correctly, you are using
> >>> synchronous calls. To get high throughput values, you need multiple
> >>> outstanding requests, so you will need to use asynchronous calls.
> >>>
> >>> -Flavio
> >>>
> >>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
> >>>
> >>> P.S. 1 we use zookeeper 3.3.2
> >>> P.S. 2 all our testing process get data from the same znode. The size
> of
> >>> data on the znode is less than 1K.
> >>>
> >>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
> >>>
> >>> Hi all:
> >>>
> >>>
> >>> These days my friend and I did some performance tests on zookeeper. We
> >>>
> >>> found the performance of zookeeper is not as good as it is described in
> >>> the
> >>>
> >>> Zookeeper Overview (
> >>>
> >>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) .
> In
> >>>
> >>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write
> Ratio
> >>>
> >>> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput
> >>> can
> >>>
> >>> reach about 80000, if the requests are all reads. However, we cannot
> get
> >>>
> >>> results like that in our performance test with the synchronized
> >>> interface,
> >>>
> >>> zkpython.
> >>>
> >>>
> >>> Here is some of our test results:
> >>>
> >>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
> >>>
> >>>
> >>> § 1 client server，1 process per client server，connect 1 zookeeper
> >>>
> >>> server，all reads：cpu：8%~9%，qps：2208，latency：0.000453s
> >>>
> >>> § 1 client server，1 process per client server，connect all 3 zookeeper

Sorry for the report, I made a mistake. When I turn off the debug level log
in zookeeper servers, the performance is like you described.
Thanks for you guys.
在 2011年3月10日 上午10:31，lei he <[EMAIL PROTECTED]>写道：

> thanks for reply. I use scp to check my ethernet again, the transmission
> rate is about 60MB/s. I send about 100000 requests in each process(60
> processes in 2 clients), and it costs about 550 seconds to finish all the
> requests. And I am sure that all the processes had been spread around all
> the servers.
>
> Thanks
> Lei, He
>
> 在 2011年3月9日 上午9:22，Benjamin Reed <[EMAIL PROTECTED]>写道：
>
> are you using 1 gbs ethernet or 100 mbs? those numbers seem to correspond
>> to
>> 100 mbs. you may want to do a simple bandwidth test just to make sure.
>>
>> how long are you running the test for?
>>
>> also, you might check that the clients are being spread around the zk
>> servers. using the 4 letter works or phunts zktop.
>>
>> ben
>>
>> 2011/3/9 lei he <[EMAIL PROTECTED]>
>>
>> > Hi Flavio and Qian, I did some tests in my environment, too. And it
>> seems
>> > it is not as good as it supposed to be. I used async
>> interface(zoo_aget),
>> > send about 10000 requests to3  servers at the same time. But it turn out
>> to
>> > be on the same level with Qian's test. Is there anything I should pay
>> > attentio to?
>> >
>> > thanks a lot
>> > He, Lei
>> >
>> > 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>> >
>> >> How many outstanding requests do you have at a time, Qian? Only one? If
>> >> so, you should instead have multiple outstanding. That is, you don't
>> wait
>> >> for the callback to issue a new request.
>> >>
>> >> -Flavio
>> >>
>> >> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>> >>
>> >> Hi Flavio, asynchronous calls doesn't perform better, here is some
>> results
>> >> we've got,
>> >>
>> >> § 1 client server，1 process per client server，connect 1 zookeeper
>> >> server，all reads：cpu：14%~15%，qps：3833，latency：0.000261
>> >> § 1 client server，1 process per client server，connect all 3 zookeeper
>> >> server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
>> >> § 1 client server，10 process per client server，connect all 3 zookeeper
>> >> server，all reads，cpu：13%~20%，qps：14000->12000，latency：0.000469
>> >> *§ 1 client server，30 process per client server，connect all 3 zookeeper
>> >> server，all reads，cpu：15%~20%，qps：14000->10000，,latency：
>> >> § 2 client server，30 process per client server，connect all 3 zookeeper
>> >> server，all reads，cpu：15%~20%，qps：about 11000，latency：*
>> >>
>> >> It seems that the asynchronous calls perform even worse than the
>> >> synchronous calls.
>> >>
>> >>
>> >> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]
>> >wrote:
>> >>
>> >>> Hi Qian, If I understand your description correctly, you are using
>> >>> synchronous calls. To get high throughput values, you need multiple
>> >>> outstanding requests, so you will need to use asynchronous calls.
>> >>>
>> >>> -Flavio
>> >>>
>> >>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>> >>>
>> >>> P.S. 1 we use zookeeper 3.3.2
>> >>> P.S. 2 all our testing process get data from the same znode. The size
>> of
>> >>> data on the znode is less than 1K.
>> >>>
>> >>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]>
>> wrote:
>> >>>
>> >>> Hi all:
>> >>>
>> >>>
>> >>> These days my friend and I did some performance tests on zookeeper. We
>> >>>
>> >>> found the performance of zookeeper is not as good as it is described
>> in
>> >>> the
>> >>>
>> >>> Zookeeper Overview (
>> >>>
>> >>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) .
>> In
>> >>>
>> >>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write
>> Ratio
>> >>>
>> >>> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput
>> >>> can
>> >>>
>> >>> reach about 80000, if the requests are all reads. However, we cannot
>> get
>> >>>
>> >>> results like that in our performance test with the synchronized
>> >>> interface,
>> >>>
�atency：0.000764s
�atency：*
Best Regards
Lei, He

Ah, right! Good catch.

-Flavio

On Mar 21, 2011, at 6:35 AM, lei he wrote:

> Sorry for the report, I made a mistake. When I turn off the debug  
> level log
> in zookeeper servers, the performance is like you described.
> Thanks for you guys.
> 在 2011年3月10日 上午10:31，lei he <[EMAIL PROTECTED]>写道：
>
>> thanks for reply. I use scp to check my ethernet again, the  
>> transmission
>> rate is about 60MB/s. I send about 100000 requests in each process(60
>> processes in 2 clients), and it costs about 550 seconds to finish  
>> all the
>> requests. And I am sure that all the processes had been spread  
>> around all
>> the servers.
>>
>> Thanks
>> Lei, He
>>
>> 在 2011年3月9日 上午9:22，Benjamin Reed <[EMAIL PROTECTED]>写道：
>>
>> are you using 1 gbs ethernet or 100 mbs? those numbers seem to  
>> correspond
>>> to
>>> 100 mbs. you may want to do a simple bandwidth test just to make  
>>> sure.
>>>
>>> how long are you running the test for?
>>>
>>> also, you might check that the clients are being spread around the  
>>> zk
>>> servers. using the 4 letter works or phunts zktop.
>>>
>>> ben
>>>
>>> 2011/3/9 lei he <[EMAIL PROTECTED]>
>>>
>>>> Hi Flavio and Qian, I did some tests in my environment, too. And it
>>> seems
>>>> it is not as good as it supposed to be. I used async
>>> interface(zoo_aget),
>>>> send about 10000 requests to3  servers at the same time. But it  
>>>> turn out
>>> to
>>>> be on the same level with Qian's test. Is there anything I should  
>>>> pay
>>>> attentio to?
>>>>
>>>> thanks a lot
>>>> He, Lei
>>>>
>>>> 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>>>>
>>>>> How many outstanding requests do you have at a time, Qian? Only  
>>>>> one? If
>>>>> so, you should instead have multiple outstanding. That is, you  
>>>>> don't
>>> wait
>>>>> for the callback to issue a new request.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>>>>>
>>>>> Hi Flavio, asynchronous calls doesn't perform better, here is some
>>> results
>>>>> we've got,
>>>>>
>>>>> § 1 client server，1 process per client server，connect 1  
>>>>> zookeeper
>>>>> server，all reads：cpu：14%~15%，qps：3833，latency：0.000261
>>>>> § 1 client server，1 process per client server，connect all 3  
>>>>> zookeeper
>>>>> server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
>>>>> § 1 client server，10 process per client server，connect all 3  
>>>>> zookeeper
>>>>> server，all reads，cpu：13%~20%，qps：14000->12000，latency：
>>>>> 0.000469
>>>>> *§ 1 client server，30 process per client server，connect all 3  
>>>>> zookeeper
>>>>> server，all reads，cpu：15%~20%，qps：14000->10000，,latency：
>>>>> § 2 client server，30 process per client server，connect all 3  
>>>>> zookeeper
>>>>> server，all reads，cpu：15%~20%，qps：about 11000，latency：*
>>>>>
>>>>> It seems that the asynchronous calls perform even worse than the
>>>>> synchronous calls.
>>>>>
>>>>>
>>>>> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]
>>>> wrote:
>>>>>
>>>>>> Hi Qian, If I understand your description correctly, you are  
>>>>>> using
>>>>>> synchronous calls. To get high throughput values, you need  
>>>>>> multiple
>>>>>> outstanding requests, so you will need to use asynchronous calls.
>>>>>>
>>>>>> -Flavio
>>>>>>
>>>>>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>>>>>>
>>>>>> P.S. 1 we use zookeeper 3.3.2
>>>>>> P.S. 2 all our testing process get data from the same znode.  
>>>>>> The size
>>> of
>>>>>> data on the znode is less than 1K.
>>>>>>
>>>>>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]>
>>> wrote:
>>>>>>
>>>>>> Hi all:
>>>>>>
>>>>>>
>>>>>> These days my friend and I did some performance tests on  
>>>>>> zookeeper. We
>>>>>>
>>>>>> found the performance of zookeeper is not as good as it is  
>>>>>> described
>>> in
>>>>>> the
>>>>>>
>>>>>> Zookeeper Overview (
>>>>>>
>>>>>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html)
>>>>>>  .
>>> In
>>>>>>
>>>>>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-
>>>>>> Write
>
flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301

really? maybe that's the reason for my bad performance results too. However,
how could logging makes so huge effect to the performance. I mean, usually
it only leads to about 20% performance loss when you adding more log in the
program. This phenomenon is beyond my understanding. Could any one give me
some explanation?

thanks

2011/3/21 Flavio Junqueira <[EMAIL PROTECTED]>

> Ah, right! Good catch.
>
> -Flavio
>
> On Mar 21, 2011, at 6:35 AM, lei he wrote:
>
> Sorry for the report, I made a mistake. When I turn off the debug level log
> in zookeeper servers, the performance is like you described.
> Thanks for you guys.
> 在 2011年3月10日 上午10:31���lei he <[EMAIL PROTECTED]>写道：
>
> thanks for reply. I use scp to check my ethernet again, the transmission
>
> rate is about 60MB/s. I send about 100000 requests in each process(60
>
> processes in 2 clients), and it costs about 550 seconds to finish all the
>
> requests. And I am sure that all the processes had been spread around all
>
> the servers.
>
>
> Thanks
>
> Lei, He
>
>
> 在 2011年3月9日 上午9:22��Benjamin Reed <[EMAIL PROTECTED]>写道：
>
>
> are you using 1 gbs ethernet or 100 mbs? those numbers seem to correspond
>
> to
>
> 100 mbs. you may want to do a simple bandwidth test just to make sure.
>
>
> how long are you running the test for?
>
>
> also, you might check that the clients are being spread around the zk
>
> servers. using the 4 letter works or phunts zktop.
>
>
> ben
>
>
> 2011/3/9 lei he <[EMAIL PROTECTED]>
>
>
> Hi Flavio and Qian, I did some tests in my environment, too. And it
>
> seems
>
> it is not as good as it supposed to be. I used async
>
> interface(zoo_aget),
>
> send about 10000 requests to3  servers at the same time. But it turn out
>
> to
>
> be on the same level with Qian's test. Is there anything I should pay
>
> attentio to?
>
>
> thanks a lot
>
> He, Lei
>
>
> 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>
>
> How many outstanding requests do you have at a time, Qian? Only one? If
>
> so, you should instead have multiple outstanding. That is, you don't
>
> wait
>
> for the callback to issue a new request.
>
>
> -Flavio
>
>
> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>
>
> Hi Flavio, asynchronous calls doesn't perform better, here is some
>
> results
>
> we've got,
>
>
> § 1 client server，1 process per client server，connect 1 zookeeper
>
> server，all reads：cpu：14%~15%，qps：3833，latency：0.000261
>
> § 1 client server，1 process per client server，connect all 3 zookeeper
>
> server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
>
> § 1 client server，10 process per client server，connect all 3 zookeeper
>
> server，all reads，cpu：13%~20%，qps：14000->12000，latency：0.000469
>
> *§ 1 client server，30 process per client server，connect all 3 zookeeper
>
> server，all reads，cpu：15%~20%，qps：14000->10000，,latency：
>
> § 2 client server，30 process per client server，connect all 3 zookeeper
>
> server，all reads，cpu：15%~20%，qps：about 11000，latency：*
>
>
> It seems that the asynchronous calls perform even worse than the
>
> synchronous calls.
>
>
>
> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]
>
> wrote:
>
>
> Hi Qian, If I understand your description correctly, you are using
>
> synchronous calls. To get high throughput values, you need multiple
>
> outstanding requests, so you will need to use asynchronous calls.
>
>
> -Flavio
>
>
> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>
>
> P.S. 1 we use zookeeper 3.3.2
>
> P.S. 2 all our testing process get data from the same znode. The size
>
> of
>
> data on the znode is less than 1K.
>
>
> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]>
>
> wrote:
>
>
> Hi all:
>
>
>
> These days my friend and I did some performance tests on zookeeper. We
>
>
> found the performance of zookeeper is not as good as it is described
>
> in
>
> the
>
>
> Zookeeper Overview (
>
>
> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) .
>
> In
>
>
> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write
With Regards!

Ye, Qian

Qian, I'm not sure where you're getting 20% performance degradation  
from, but I think multiple people on this list have observed severe  
performance degradation due to logging. I don't have numbers at this  
time to share with you, though. Perhaps Lei He can share some more  
numbers to help you understand the exact effect.

-Flavio

On Mar 21, 2011, at 1:14 PM, Qian Ye wrote:

> really? maybe that's the reason for my bad performance results too.  
> However, how could logging makes so huge effect to the performance.  
> I mean, usually it only leads to about 20% performance loss when you  
> adding more log in the program. This phenomenon is beyond my  
> understanding. Could any one give me some explanation?
>
> thanks
>
> 2011/3/21 Flavio Junqueira <[EMAIL PROTECTED]>
> Ah, right! Good catch.
>
> -Flavio
>
> On Mar 21, 2011, at 6:35 AM, lei he wrote:
>
>> Sorry for the report, I made a mistake. When I turn off the debug  
>> level log
>> in zookeeper servers, the performance is like you described.
>> Thanks for you guys.
>> 在 2011年3月10日 上午10:31���lei he <[EMAIL PROTECTED]>写道：
>>
>>> thanks for reply. I use scp to check my ethernet again, the  
>>> transmission
>>> rate is about 60MB/s. I send about 100000 requests in each  
>>> process(60
>>> processes in 2 clients), and it costs about 550 seconds to finish  
>>> all the
>>> requests. And I am sure that all the processes had been spread  
>>> around all
>>> the servers.
>>>
>>> Thanks
>>> Lei, He
>>>
>>> 在 2011年3月9日 上午9:22��Benjamin Reed <[EMAIL PROTECTED]>写
>>> 道：
>>>
>>> are you using 1 gbs ethernet or 100 mbs? those numbers seem to  
>>> correspond
>>>> to
>>>> 100 mbs. you may want to do a simple bandwidth test just to make  
>>>> sure.
>>>>
>>>> how long are you running the test for?
>>>>
>>>> also, you might check that the clients are being spread around  
>>>> the zk
>>>> servers. using the 4 letter works or phunts zktop.
>>>>
>>>> ben
>>>>
>>>> 2011/3/9 lei he <[EMAIL PROTECTED]>
>>>>
>>>>> Hi Flavio and Qian, I did some tests in my environment, too. And  
>>>>> it
>>>> seems
>>>>> it is not as good as it supposed to be. I used async
>>>> interface(zoo_aget),
>>>>> send about 10000 requests to3  servers at the same time. But it  
>>>>> turn out
>>>> to
>>>>> be on the same level with Qian's test. Is there anything I  
>>>>> should pay
>>>>> attentio to?
>>>>>
>>>>> thanks a lot
>>>>> He, Lei
>>>>>
>>>>> 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>>>>>
>>>>>> How many outstanding requests do you have at a time, Qian? Only  
>>>>>> one? If
>>>>>> so, you should instead have multiple outstanding. That is, you  
>>>>>> don't
>>>> wait
>>>>>> for the callback to issue a new request.
>>>>>>
>>>>>> -Flavio
>>>>>>
>>>>>> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>>>>>>
>>>>>> Hi Flavio, asynchronous calls doesn't perform better, here is  
>>>>>> some
>>>> results
>>>>>> we've got,
>>>>>>
>>>>>> § 1 client server，1 process per client server，connect 1  
>>>>>> zookeeper
>>>>>> server，all reads：cpu：14%~15%，qps：3833，latency：
>>>>>> 0.000261
>>>>>> § 1 client server，1 process per client server，connect all 3  
>>>>>> zookeeper
>>>>>> server，all reads：cpu：14%~15%，qps：3832，latency：
>>>>>> 0.000261
>>>>>> § 1 client server，10 process per client server，connect all  
>>>>>> 3 zookeeper
>>>>>> server，all reads，cpu：13%~20%，qps���14000-
>>>>>> >12000，latency：0.000469
>>>>>> *§ 1 client server，30 process per client server，connect all  
>>>>>> 3 zookeeper
>>>>>> server，all reads，cpu：15%~20%，qps���14000-
>>>>>> >10000，,latency：
>>>>>> § 2 client server，30 process per client server，connect all  
>>>>>> 3 zookeeper
>>>>>> server，all reads，cpu：15%~20%，qps���about  
>>>>>> 11000，latency：*
>>>>>>
>>>>>> It seems that the asynchronous calls perform even worse than the
>>>>>> synchronous calls.
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]
>>>>> wrote:
>>>>>>
>>>>>>> Hi Qian, If I understand your description correctly, you are  
>>>>>>> using
>>
flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301

Hi Flavio, when I use asynchronous calls, i won't wait for the callback to
issue a new request. Could you tell me the test method you used in your
test. What is the CPU utilization rate when the perforance reach the
maximum?

On Wed, Mar 9, 2011 at 5:28 PM, Flavio Junqueira <[EMAIL PROTECTED]> wrote:

> How many outstanding requests do you have at a time, Qian? Only one? If so,
> you should instead have multiple outstanding. That is, you don't wait for
> the callback to issue a new request.
>
> -Flavio
>
> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>
> Hi Flavio, asynchronous calls doesn't perform better, here is some results
> we've got,
>
> § 1 client server，1 process per client server，connect 1 zookeeper
> server，all reads：cpu：14%~15%，qps：3833，latency：0.000261
> § 1 client server，1 process per client server，connect all 3 zookeeper
> server，all reads：cpu：14%~15%，qps：3832，latency：0.000261
> § 1 client server，10 process per client server，connect all 3 zookeeper
> server，all reads，cpu：13%~20%，qps：14000->12000，latency：0.000469
> *§ 1 client server，30 process per client server，connect all 3 zookeeper
> server，all reads，cpu：15%~20%，qps：14000->10000，,latency：
> § 2 client server，30 process per client server，connect all 3 zookeeper
> server，all reads，cpu：15%~20%，qps：about 11000，latency：*
>
> It seems that the asynchronous calls perform even worse than the
> synchronous calls.
>
>
> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]>wrote:
>
>> Hi Qian, If I understand your description correctly, you are using
>> synchronous calls. To get high throughput values, you need multiple
>> outstanding requests, so you will need to use asynchronous calls.
>>
>> -Flavio
>>
>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>>
>> P.S. 1 we use zookeeper 3.3.2
>> P.S. 2 all our testing process get data from the same znode. The size of
>> data on the znode is less than 1K.
>>
>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>>
>> Hi all:
>>
>>
>> These days my friend and I did some performance tests on zookeeper. We
>>
>> found the performance of zookeeper is not as good as it is described in
>> the
>>
>> Zookeeper Overview (
>>
>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html) . In
>>
>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write Ratio
>>
>> Varies" shows that in a ensemble of 3 Zookeeper server, the throughput can
>>
>> reach about 80000, if the requests are all reads. However, we cannot get
>>
>> results like that in our performance test with the synchronized interface,
>>
>> zkpython.
>>
>>
>> Here is some of our test results:
>>
>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>
>>
>> § 1 client server，1 process per client server，connect 1 zookeeper
>>
>> server，all reads：cpu：8%~9%，qps：2208，latency：0.000453s
>>
>> § 1 client server，1 process per client server，connect all 3 zookeeper
>>
>> server，all reads：cpu：8%~9%，qps：2376.241573 ，latency:0.000421s
>>
>> § 1 client server，1 process per client server，connect all 3 zookeeper
>>
>> server，all reads，cpu：10%~20%，qps��15600，latency：0.000764s
>>
>> *§ 1 client server，30 process per client server，connect all 3 zookeeper
>>
>> server，all reads，cpu：10%~20%，qps��15200，latency：*
>>
>> *§ 2 client server，30 process **per client server**��connect all 3
>>
>> zookeeper server，all reads，cpu：10%~20%，qps：15800，latency：0.003487*
>>
>>
>> qps means "query per second", that is throughput. The result shows that
>>
>> when adding more client server, the utilization rate of CPU don't
>> increase,
>>
>> and the throughput don't increase much. It seems that the throughput won't
>>
>> reach 80000, even if we add 28 more client servers to reach the number you
>>
>> mentioned in the Zookeeper Overview.
>>
>>
>> Maybe I've done the tests wrong. Is there any particular thing I should
>> pay
>>
>> attention to in this case? We set the max java heap size to 12GB in our
>>
>> test.
>>
>>
>> *Could you tell me the details about how you do the performance test, from
With Regards!

Ye, Qian