Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Zookeeper in Kafka has gets a very low Performance


+
sun liwei 2012-09-12, 10:07
+
sun liwei 2012-09-12, 10:33
+
Kathleen Ting 2012-09-12, 18:47
+
sun liwei 2012-09-13, 01:20
+
sun liwei 2012-09-13, 01:44
Copy link to this message
-
Re: Zookeeper in Kafka has gets a very low Performance
Maybe it has something to do with the frequent writes delaying reads ?
If a write is submitted to a follower and then a read (getData),
no matter if the write and the read are to the same data, the read
will block until the write completes. That's what ZOOKEEPER-1505 tries
to address.
Alex
On Wed, Sep 12, 2012 at 6:44 PM, sun liwei <[EMAIL PROTECTED]> wrote:
> Let me give more details:
> I have two zookeeper clusters: A(3.3.4) and B(3.4.3).
> A is used by Kafka only. B servers other application rather than Kafka.
> The performance of A is very bad no matter which path I read with getData;
> But B works well.
>
> My suspicion is that Kafka makes A slowing down. There are 1000+
> consumers(each consumer is a node) in A. Kafka updates the value of all
> these consumers very frequently. So, that means the write operations in A
> is much more than read operations.
>
>
> On Wed, Sep 12, 2012 at 6:33 PM, sun liwei <[EMAIL PROTECTED]> wrote:
>
>> The version of zookeeper is 3.3.4.
>>
>>
>> On Wed, Sep 12, 2012 at 6:07 PM, sun liwei <[EMAIL PROTECTED]> wrote:
>>
>>> Here is what I have:
>>>
>>>
>>>    - zookeeper cluster with 3 severs
>>>    - 1700+ kafka topics (that means there are 1700+ children under path
>>>    '/brokers/topics'), these topics are written frequently by kafka.
>>>    - log file size is 67108880 byte,  snapshot file size is 4192072
>>>    byte, new log/snapshot files are created every one or two minutes
>>>
>>> The problem is that it takes more than 40ms to getData of a single topic.
>>> I have another zookeeper cluster and the getData takes no more than 2ms to
>>> getData of a single path. This cluster doesn't server kafka.
>>>
>>> Any body knows why the performance is so bad?
>>>
>>> Many thanks and best regards.
>>>
>>> Liwei
>>>
>>
>>
+
sun liwei 2012-09-13, 06:47
+
Alexander Shraer 2012-09-13, 13:44
+
Alexander Shraer 2012-09-13, 14:08
+
Kathleen Ting 2012-09-13, 17:21
+
Henry Robinson 2012-09-12, 18:49
+
sun liwei 2012-09-13, 01:24