Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper, mail # dev - Something about performance of Zookeeper


Copy link to this message
-
Re: Something about performance of Zookeeper
Flavio Junqueira 2011-03-08, 16:29
Hi Qian, If I understand your description correctly, you are using  
synchronous calls. To get high throughput values, you need multiple  
outstanding requests, so you will need to use asynchronous calls.

-Flavio

On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:

> P.S. 1 we use zookeeper 3.3.2
> P.S. 2 all our testing process get data from the same znode. The  
> size of
> data on the znode is less than 1K.
>
> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]> wrote:
>
>> Hi all:
>>
>> These days my friend and I did some performance tests on zookeeper.  
>> We
>> found the performance of zookeeper is not as good as it is  
>> described in the
>> Zookeeper Overview (
>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/
>> zookeeperOver.html) . In
>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-Write  
>> Ratio
>> Varies" shows that in a ensemble of 3 Zookeeper server, the  
>> throughput can
>> reach about 80000, if the requests are all reads. However, we  
>> cannot get
>> results like that in our performance test with the synchronized  
>> interface,
>> zkpython.
>>
>> Here is some of our test results:
>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>
>> § 1 client server,1 process per client server,connect 1  
>> zookeeper
>> server,all reads:cpu:8%~9%,qps:2208,latency���0.000453s
>> § 1 client server,1 process per client server,connect all 3  
>> zookeeper
>> server,all reads:cpu:8%~9%,qps:2376.241573 ,latency:
>> 0.000421s
>> § 1 client server,1 process per client server,connect all 3  
>> zookeeper
>> server,all reads,cpu:10%~20%,qps:15600,latency:
>> 0.000764s
>> *§ 1 client server,30 process per client server,connect all 3  
>> zookeeper
>> server,all reads,cpu:10%~20%,qps:15200,latency:*
>> *§ 2 client server,30 process **per client server**,connect all  
>> 3
>> zookeeper server,all reads,cpu:10%~20%,qps:
>> 15800,latency:0.003487*
>>
>> qps means "query per second", that is throughput. The result shows  
>> that
>> when adding more client server, the utilization rate of CPU don't  
>> increase,
>> and the throughput don't increase much. It seems that the  
>> throughput won't
>> reach 80000, even if we add 28 more client servers to reach the  
>> number you
>> mentioned in the Zookeeper Overview.
>>
>> Maybe I've done the tests wrong. Is there any particular thing I  
>> should pay
>> attention to in this case? We set the max java heap size to 12GB in  
>> our
>> test.
>>
>> *Could you tell me the details about how you do the performance  
>> test, from
>> which you get the results showed in the Zookeeper Overview?*
>>
>> --
>> With Regards!
>>
>> Ye, Qian
>>
>>
>
>
> --
> With Regards!
>
> Ye, Qian

flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301