Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # dev >> Something about performance of Zookeeper


+
Qian Ye 2011-03-08, 16:08
+
Qian Ye 2011-03-08, 16:16
+
Flavio Junqueira 2011-03-08, 16:29
+
Qian Ye 2011-03-09, 02:44
+
Eugene Koontz 2011-03-09, 07:24
+
Flavio Junqueira 2011-03-09, 09:28
+
lei he 2011-03-09, 13:00
+
Benjamin Reed 2011-03-09, 17:22
+
lei he 2011-03-10, 02:31
+
lei he 2011-03-21, 05:35
Copy link to this message
-
Re: Something about performance of Zookeeper
Ah, right! Good catch.

-Flavio

On Mar 21, 2011, at 6:35 AM, lei he wrote:

> Sorry for the report, I made a mistake. When I turn off the debug  
> level log
> in zookeeper servers, the performance is like you described.
> Thanks for you guys.
> 在 2011年3月10日 上午10:31,lei he <[EMAIL PROTECTED]>写道:
>
>> thanks for reply. I use scp to check my ethernet again, the  
>> transmission
>> rate is about 60MB/s. I send about 100000 requests in each process(60
>> processes in 2 clients), and it costs about 550 seconds to finish  
>> all the
>> requests. And I am sure that all the processes had been spread  
>> around all
>> the servers.
>>
>> Thanks
>> Lei, He
>>
>> 在 2011年3月9日 上午9:22,Benjamin Reed <[EMAIL PROTECTED]>写道:
>>
>> are you using 1 gbs ethernet or 100 mbs? those numbers seem to  
>> correspond
>>> to
>>> 100 mbs. you may want to do a simple bandwidth test just to make  
>>> sure.
>>>
>>> how long are you running the test for?
>>>
>>> also, you might check that the clients are being spread around the  
>>> zk
>>> servers. using the 4 letter works or phunts zktop.
>>>
>>> ben
>>>
>>> 2011/3/9 lei he <[EMAIL PROTECTED]>
>>>
>>>> Hi Flavio and Qian, I did some tests in my environment, too. And it
>>> seems
>>>> it is not as good as it supposed to be. I used async
>>> interface(zoo_aget),
>>>> send about 10000 requests to3  servers at the same time. But it  
>>>> turn out
>>> to
>>>> be on the same level with Qian's test. Is there anything I should  
>>>> pay
>>>> attentio to?
>>>>
>>>> thanks a lot
>>>> He, Lei
>>>>
>>>> 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>>>>
>>>>> How many outstanding requests do you have at a time, Qian? Only  
>>>>> one? If
>>>>> so, you should instead have multiple outstanding. That is, you  
>>>>> don't
>>> wait
>>>>> for the callback to issue a new request.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>>>>>
>>>>> Hi Flavio, asynchronous calls doesn't perform better, here is some
>>> results
>>>>> we've got,
>>>>>
>>>>> § 1 client server,1 process per client server,connect 1  
>>>>> zookeeper
>>>>> server,all reads:cpu:14%~15%,qps:3833,latency:0.000261
>>>>> § 1 client server,1 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads:cpu:14%~15%,qps:3832,latency:0.000261
>>>>> § 1 client server,10 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads,cpu:13%~20%,qps:14000->12000,latency:
>>>>> 0.000469
>>>>> *§ 1 client server,30 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads,cpu:15%~20%,qps:14000->10000,,latency:
>>>>> § 2 client server,30 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads,cpu:15%~20%,qps:about 11000,latency:*
>>>>>
>>>>> It seems that the asynchronous calls perform even worse than the
>>>>> synchronous calls.
>>>>>
>>>>>
>>>>> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]
>>>> wrote:
>>>>>
>>>>>> Hi Qian, If I understand your description correctly, you are  
>>>>>> using
>>>>>> synchronous calls. To get high throughput values, you need  
>>>>>> multiple
>>>>>> outstanding requests, so you will need to use asynchronous calls.
>>>>>>
>>>>>> -Flavio
>>>>>>
>>>>>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>>>>>>
>>>>>> P.S. 1 we use zookeeper 3.3.2
>>>>>> P.S. 2 all our testing process get data from the same znode.  
>>>>>> The size
>>> of
>>>>>> data on the znode is less than 1K.
>>>>>>
>>>>>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]>
>>> wrote:
>>>>>>
>>>>>> Hi all:
>>>>>>
>>>>>>
>>>>>> These days my friend and I did some performance tests on  
>>>>>> zookeeper. We
>>>>>>
>>>>>> found the performance of zookeeper is not as good as it is  
>>>>>> described
>>> in
>>>>>> the
>>>>>>
>>>>>> Zookeeper Overview (
>>>>>>
>>>>>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html)
>>>>>>  .
>>> In
>>>>>>
>>>>>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-
>>>>>> Write
>
flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301
+
Qian Ye 2011-03-21, 12:14
+
Flavio Junqueira 2011-03-21, 12:23
+
Qian Ye 2011-03-09, 15:36
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB