Has any performance testing been done on !ZooKeeper to determine how many
concurrent watches are supported? The application I am considering using ZK
for would use ~4000000 znodes and set ~4000000 concurrent watches.  All the
performance numbers I found on the ZK docs were reads/write per second. Are
there any numbers for concurrent watches?

I believe that folks at FB have done some very very heavy-load ZK usage and
could talk more about scale, but I suspect that the current releases of ZK
would not scale to what you are looking for. Can you tell us more details
about what you're trying to do?

C

On Mon, Oct 15, 2012 at 9:23 PM, Rohan Desai <[EMAIL PROTECTED]>wrote:

> Has any performance testing been done on !ZooKeeper to determine how many
> concurrent watches are supported? The application I am considering using ZK
> for would use ~4000000 znodes and set ~4000000 concurrent watches.  All the
> performance numbers I found on the ZK docs were reads/write per second. Are
> there any numbers for concurrent watches?
>

It's been a while, and it's probably highly dependent on the exact use
cases, but I have tested successfully with 5m znodes and 25m watches.
This was a three server ensemble. In this case 500 clients each
created 10k znodes, then set 5 watches on each of the znodes they
created, then deleted the znodes. The watches were notified on order
of seconds. I did have to tune the amount of memory and GC but otw it
worked fine for me.

Patrick

On Mon, Oct 15, 2012 at 6:31 PM, Camille Fournier <[EMAIL PROTECTED]> wrote:
> I believe that folks at FB have done some very very heavy-load ZK usage and
> could talk more about scale, but I suspect that the current releases of ZK
> would not scale to what you are looking for. Can you tell us more details
> about what you're trying to do?
>
> C
>
> On Mon, Oct 15, 2012 at 9:23 PM, Rohan Desai <[EMAIL PROTECTED]>wrote:
>
>> Has any performance testing been done on !ZooKeeper to determine how many
>> concurrent watches are supported? The application I am considering using ZK
>> for would use ~4000000 znodes and set ~4000000 concurrent watches.  All the
>> performance numbers I found on the ZK docs were reads/write per second. Are
>> there any numbers for concurrent watches?
>>

You can add more observers to scale the number of watches that the
ensemble can handle.

Znode count will depend on how much RAM you have and how much data are in
each znode. Our last benchmark shows that empty znode consume 600 bytes of
memory.  

--
Thawan Kooburat

On 10/15/12 11:43 PM, "Patrick Hunt" <[EMAIL PROTECTED]> wrote:

>It's been a while, and it's probably highly dependent on the exact use
>cases, but I have tested successfully with 5m znodes and 25m watches.
>This was a three server ensemble. In this case 500 clients each
>created 10k znodes, then set 5 watches on each of the znodes they
>created, then deleted the znodes. The watches were notified on order
>of seconds. I did have to tune the amount of memory and GC but otw it
>worked fine for me.
>
>Patrick
>
>On Mon, Oct 15, 2012 at 6:31 PM, Camille Fournier <[EMAIL PROTECTED]>
>wrote:
>> I believe that folks at FB have done some very very heavy-load ZK usage
>>and
>> could talk more about scale, but I suspect that the current releases of
>>ZK
>> would not scale to what you are looking for. Can you tell us more
>>details
>> about what you're trying to do?
>>
>> C
>>
>> On Mon, Oct 15, 2012 at 9:23 PM, Rohan Desai
>><[EMAIL PROTECTED]>wrote:
>>
>>> Has any performance testing been done on !ZooKeeper to determine how
>>>many
>>> concurrent watches are supported? The application I am considering
>>>using ZK
>>> for would use ~4000000 znodes and set ~4000000 concurrent watches.
>>>All the
>>> performance numbers I found on the ZK docs were reads/write per
>>>second. Are
>>> there any numbers for concurrent watches?
>>>