Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper, mail # dev - Something about performance of Zookeeper


+
Qian Ye 2011-03-08, 16:08
+
Qian Ye 2011-03-08, 16:16
+
Flavio Junqueira 2011-03-08, 16:29
+
Qian Ye 2011-03-09, 02:44
+
Eugene Koontz 2011-03-09, 07:24
+
Flavio Junqueira 2011-03-09, 09:28
+
lei he 2011-03-09, 13:00
+
Benjamin Reed 2011-03-09, 17:22
+
lei he 2011-03-10, 02:31
+
lei he 2011-03-21, 05:35
Copy link to this message
-
Re: Something about performance of Zookeeper
Flavio Junqueira 2011-03-21, 10:50
Ah, right! Good catch.

-Flavio

On Mar 21, 2011, at 6:35 AM, lei he wrote:

> Sorry for the report, I made a mistake. When I turn off the debug  
> level log
> in zookeeper servers, the performance is like you described.
> Thanks for you guys.
> 在 2011年3月10日 上午10:31,lei he <[EMAIL PROTECTED]>写道:
>
>> thanks for reply. I use scp to check my ethernet again, the  
>> transmission
>> rate is about 60MB/s. I send about 100000 requests in each process(60
>> processes in 2 clients), and it costs about 550 seconds to finish  
>> all the
>> requests. And I am sure that all the processes had been spread  
>> around all
>> the servers.
>>
>> Thanks
>> Lei, He
>>
>> 在 2011年3月9日 上午9:22,Benjamin Reed <[EMAIL PROTECTED]>写道:
>>
>> are you using 1 gbs ethernet or 100 mbs? those numbers seem to  
>> correspond
>>> to
>>> 100 mbs. you may want to do a simple bandwidth test just to make  
>>> sure.
>>>
>>> how long are you running the test for?
>>>
>>> also, you might check that the clients are being spread around the  
>>> zk
>>> servers. using the 4 letter works or phunts zktop.
>>>
>>> ben
>>>
>>> 2011/3/9 lei he <[EMAIL PROTECTED]>
>>>
>>>> Hi Flavio and Qian, I did some tests in my environment, too. And it
>>> seems
>>>> it is not as good as it supposed to be. I used async
>>> interface(zoo_aget),
>>>> send about 10000 requests to3  servers at the same time. But it  
>>>> turn out
>>> to
>>>> be on the same level with Qian's test. Is there anything I should  
>>>> pay
>>>> attentio to?
>>>>
>>>> thanks a lot
>>>> He, Lei
>>>>
>>>> 2011/3/9 Flavio Junqueira <[EMAIL PROTECTED]>
>>>>
>>>>> How many outstanding requests do you have at a time, Qian? Only  
>>>>> one? If
>>>>> so, you should instead have multiple outstanding. That is, you  
>>>>> don't
>>> wait
>>>>> for the callback to issue a new request.
>>>>>
>>>>> -Flavio
>>>>>
>>>>> On Mar 9, 2011, at 3:44 AM, Qian Ye wrote:
>>>>>
>>>>> Hi Flavio, asynchronous calls doesn't perform better, here is some
>>> results
>>>>> we've got,
>>>>>
>>>>> § 1 client server,1 process per client server,connect 1  
>>>>> zookeeper
>>>>> server,all reads:cpu:14%~15%,qps:3833,latency:0.000261
>>>>> § 1 client server,1 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads:cpu:14%~15%,qps:3832,latency:0.000261
>>>>> § 1 client server,10 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads,cpu:13%~20%,qps:14000->12000,latency:
>>>>> 0.000469
>>>>> *§ 1 client server,30 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads,cpu:15%~20%,qps:14000->10000,,latency:
>>>>> § 2 client server,30 process per client server,connect all 3  
>>>>> zookeeper
>>>>> server,all reads,cpu:15%~20%,qps:about 11000,latency:*
>>>>>
>>>>> It seems that the asynchronous calls perform even worse than the
>>>>> synchronous calls.
>>>>>
>>>>>
>>>>> On Wed, Mar 9, 2011 at 12:29 AM, Flavio Junqueira <[EMAIL PROTECTED]
>>>> wrote:
>>>>>
>>>>>> Hi Qian, If I understand your description correctly, you are  
>>>>>> using
>>>>>> synchronous calls. To get high throughput values, you need  
>>>>>> multiple
>>>>>> outstanding requests, so you will need to use asynchronous calls.
>>>>>>
>>>>>> -Flavio
>>>>>>
>>>>>> On Mar 8, 2011, at 5:16 PM, Qian Ye wrote:
>>>>>>
>>>>>> P.S. 1 we use zookeeper 3.3.2
>>>>>> P.S. 2 all our testing process get data from the same znode.  
>>>>>> The size
>>> of
>>>>>> data on the znode is less than 1K.
>>>>>>
>>>>>> On Wed, Mar 9, 2011 at 12:08 AM, Qian Ye <[EMAIL PROTECTED]>
>>> wrote:
>>>>>>
>>>>>> Hi all:
>>>>>>
>>>>>>
>>>>>> These days my friend and I did some performance tests on  
>>>>>> zookeeper. We
>>>>>>
>>>>>> found the performance of zookeeper is not as good as it is  
>>>>>> described
>>> in
>>>>>> the
>>>>>>
>>>>>> Zookeeper Overview (
>>>>>>
>>>>>> http://hadoop.apache.org/zookeeper/docs/r3.3.2/zookeeperOver.html)
>>>>>>  .
>>> In
>>>>>>
>>>>>> the Zookeeper Overview, the "ZooKeeper Throughput as the Read-
>>>>>> Write
>
flavio
junqueira

research scientist

[EMAIL PROTECTED]
direct +34 93-183-8828

avinguda diagonal 177, 8th floor, barcelona, 08018, es
phone (408) 349 3300    fax (408) 349 3301
+
Qian Ye 2011-03-21, 12:14
+
Flavio Junqueira 2011-03-21, 12:23
+
Qian Ye 2011-03-09, 15:36