Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # dev >> Something about performance of Zookeeper


+
Qian Ye 2011-03-08, 16:08
+
Qian Ye 2011-03-08, 16:16
+
Flavio Junqueira 2011-03-08, 16:29
+
Qian Ye 2011-03-09, 02:44
+
Eugene Koontz 2011-03-09, 07:24
Copy link to this message
-
Re: Something about performance of Zookeeper
How many outstanding requests do you have at a time, Qian? Only one?  
If so, you should instead have multiple outstanding. That is, you  
don't wait for the callback to issue a new request.

-Flavio

On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:

> Hi Flavio, asynchronous calls doesn't perform better, here is some  
> results we've got,
>
> § 1 client server,1 process per client server,connect 1  
> zookeeper server,all reads:cpu:14%~15%,qps:3833,latency:
> 0.000261
> § 1 client server,1 process per client server,connect all 3  
> zookeeper server,all reads:cpu:14%~15%,qps:3832,latency:
> 0.000261
> § 1 client server,10 process per client server,connect all 3  
> zookeeper server,all reads,cpu:13%~20%,qps:14000-
> >12000,latency:0.000469
> § 1 client server,30 process per client server,connect all 3  
> zookeeper server,all reads,cpu:15%~20%,qps:14000-
> >10000,,latency:
> § 2 client server,30 process per client server,connect all 3  
> zookeeper server,all reads,cpu:15%~20%,qps:about  
> 11000,latency:
>
> It seems that the asynchronous calls perform even worse than the  
> synchronous calls.
>
>
> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <fpj@yahoo-
> inc.com> wrote:
> Hi Qian, If I understand your description correctly, you are using  
> synchronous calls. To get high throughput values, you need multiple  
> outstanding requests, so you will need to use asynchronous calls.
>
> -Flavio
>
> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>
>> P.S. 1 we use zookeeper 3.3.2
>> P.S. 2 all our testing process get data from the same znode. The  
>> size of
>> data on the znode is less than 1K.
>>
>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]>  
>> wrote:
>>
>>> Hi all:
>>>
>>> These days my friend and I did some performance tests on  
>>> zookeeper. We
>>> found the performance of zookeeper is not as good as it is  
>>> described in the
>>> Zookeeper Overview (
>>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/
>>> zookeeperOver.html) . In
>>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-
>>> Write Ratio
>>> Varies" shows that in a ensemble of 3 Zookeeper server, the  
>>> throughput can
>>> reach about 80000, if the requests are all reads. However, we  
>>> cannot get
>>> results like that in our performance test with the synchronized  
>>> interface,
>>> zkpython.
>>>
>>> Here is some of our test results:
>>> (3 zookeeper ensemble, 8 core CPU,  2.4GHZ, 16 RAM, Linux 2.6.9)
>>>
>>> § 1 client server,1 process per client server,connect 1  
>>> zookeeper
>>> server,all reads:cpu:8%~9%,qps:2208,latency���0.000453s
>>> § 1 client server,1 process per client server,connect all 3  
>>> zookeeper
>>> server,all reads:cpu:8%~9%,qps:2376.241573 ,latency:
>>> 0.000421s
>>> § 1 client server,1 process per client server,connect all 3  
>>> zookeeper
>>> server,all reads,cpu:10%~20%,qps��15600,latency:
>>> 0.000764s
>>> *§ 1 client server,30 process per client server,connect all 3  
>>> zookeeper
>>> server,all reads,cpu:10%~20%,qps��15200,latency:*
>>> *§ 2 client server,30 process **per client server**,connect  
>>> all 3
>>> zookeeper server,all reads,cpu:10%~20%,qps:
>>> 15800,latency:0.003487*
>>>
>>> qps means "query per second", that is throughput. The result shows  
>>> that
>>> when adding more client server, the utilization rate of CPU don't  
>>> increase,
>>> and the throughput don't increase much. It seems that the  
>>> throughput won't
>>> reach 80000, even if we add 28 more client servers to reach the  
>>> number you
>>> mentioned in the Zookeeper Overview.
>>>
>>> Maybe I've done the tests wrong. Is there any particular thing I  
>>> should pay
>>> attention to in this case? We set the max java heap size to 12GB  
>>> in our
>>> test.
>>>
>>> *Could you tell me the details about how you do the performance  
>>> test, from
>>> which you get the results showed in the Zookeeper Overview?*
>>>
>>> --
>>> With Regards!
>>>
>>> Ye, Qian
>>>
>>>
>>
>>
>> --
>> With Regards!
>>
>> Ye, Qian

flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301
+
lei he 2011-03-09, 13:00
+
Benjamin Reed 2011-03-09, 17:22
+
lei he 2011-03-10, 02:31
+
lei he 2011-03-21, 05:35
+
Flavio Junqueira 2011-03-21, 10:50
+
Qian Ye 2011-03-21, 12:14
+
Flavio Junqueira 2011-03-21, 12:23
+
Qian Ye 2011-03-09, 15:36
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB