Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> Zookeeper in Kafka has gets a very low Performance


Copy link to this message
-
Re: Zookeeper in Kafka has gets a very low Performance
Maybe it has something to do with the frequent writes delaying reads ?
If a write is submitted to a follower and then a read (getData),
no matter if the write and the read are to the same data, the read
will block until the write completes. That's what ZOOKEEPER-1505 tries
to address.
Alex
On Wed, Sep 12, 2012 at 6:44 PM, sun liwei <[EMAIL PROTECTED]> wrote:
> Let me give more details:
> I have two zookeeper clusters: A(3.3.4) and B(3.4.3).
> A is used by Kafka only. B servers other application rather than Kafka.
> The performance of A is very bad no matter which path I read with getData;
> But B works well.
>
> My suspicion is that Kafka makes A slowing down. There are 1000+
> consumers(each consumer is a node) in A. Kafka updates the value of all
> these consumers very frequently. So, that means the write operations in A
> is much more than read operations.
>
>
> On Wed, Sep 12, 2012 at 6:33 PM, sun liwei <[EMAIL PROTECTED]> wrote:
>
>> The version of zookeeper is 3.3.4.
>>
>>
>> On Wed, Sep 12, 2012 at 6:07 PM, sun liwei <[EMAIL PROTECTED]> wrote:
>>
>>> Here is what I have:
>>>
>>>
>>>    - zookeeper cluster with 3 severs
>>>    - 1700+ kafka topics (that means there are 1700+ children under path
>>>    '/brokers/topics'), these topics are written frequently by kafka.
>>>    - log file size is 67108880 byte,  snapshot file size is 4192072
>>>    byte, new log/snapshot files are created every one or two minutes
>>>
>>> The problem is that it takes more than 40ms to getData of a single topic.
>>> I have another zookeeper cluster and the getData takes no more than 2ms to
>>> getData of a single path. This cluster doesn't server kafka.
>>>
>>> Any body knows why the performance is so bad?
>>>
>>> Many thanks and best regards.
>>>
>>> Liwei
>>>
>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB