Hi all,

I'm having a use case where there're potentially millions of client
processes connected to a Zookeeper cluster, but only a very small
proportion of them would be actively sending requests or receiving any data
(say, about 1000 of them every second) at any time. Has anyone tried this
before?

Best Regards,
Martin Kou

You'll run out of file descriptors on the ZK servers, most likely, and when
you do, the processes will behave VERY badly.
I do not recommend this if you can design some other way.

C

On Mon, Sep 17, 2012 at 6:56 PM, Martin Kou <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I'm having a use case where there're potentially millions of client
> processes connected to a Zookeeper cluster, but only a very small
> proportion of them would be actively sending requests or receiving any data
> (say, about 1000 of them every second) at any time. Has anyone tried this
> before?
>
> Best Regards,
> Martin Kou
>

The obvious solution would be running multiple Zookeepers hierarchically.
Each server would take, say, 1000 clients.  Each server would be a client
of an uber Zookeeper.  It'll increase latency some, but it scales.
On Sep 17, 2012 3:57 PM, "Martin Kou" <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I'm having a use case where there're potentially millions of client
> processes connected to a Zookeeper cluster, but only a very small
> proportion of them would be actively sending requests or receiving any data
> (say, about 1000 of them every second) at any time. Has anyone tried this
> before?
>
> Best Regards,
> Martin Kou
>

Another option is to use a proxy.

On Mon, Sep 17, 2012 at 6:44 PM, Morris Bernstein <
[EMAIL PROTECTED]> wrote:

> The obvious solution would be running multiple Zookeepers hierarchically.
> Each server would take, say, 1000 clients.  Each server would be a client
> of an uber Zookeeper.  It'll increase latency some, but it scales.
> On Sep 17, 2012 3:57 PM, "Martin Kou" <[EMAIL PROTECTED]> wrote:
>
> > Hi all,
> >
> > I'm having a use case where there're potentially millions of client
> > processes connected to a Zookeeper cluster, but only a very small
> > proportion of them would be actively sending requests or receiving any
> data
> > (say, about 1000 of them every second) at any time. Has anyone tried this
> > before?
> >
> > Best Regards,
> > Martin Kou
> >
>

Uhm, yeah, that's what I meant...
On Sep 17, 2012 7:59 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:

> Another option is to use a proxy.
>
> On Mon, Sep 17, 2012 at 6:44 PM, Morris Bernstein <
> [EMAIL PROTECTED]> wrote:
>
> > The obvious solution would be running multiple Zookeepers hierarchically.
> > Each server would take, say, 1000 clients.  Each server would be a client
> > of an uber Zookeeper.  It'll increase latency some, but it scales.
> > On Sep 17, 2012 3:57 PM, "Martin Kou" <[EMAIL PROTECTED]> wrote:
> >
> > > Hi all,
> > >
> > > I'm having a use case where there're potentially millions of client
> > > processes connected to a Zookeeper cluster, but only a very small
> > > proportion of them would be actively sending requests or receiving any
> > data
> > > (say, about 1000 of them every second) at any time. Has anyone tried
> this
> > > before?
> > >
> > > Best Regards,
> > > Martin Kou
> > >
> >
>

sorry.

Hit send a bit early.

I meant a special purpose proxy that limits the amount of notification that
goes out by changing the semantics a bit.  You really don't want millions
of change notices to go out all at once.

On Mon, Sep 17, 2012 at 8:46 PM, Morris Bernstein <
[EMAIL PROTECTED]> wrote:

> Uhm, yeah, that's what I meant...
> On Sep 17, 2012 7:59 PM, "Ted Dunning" <[EMAIL PROTECTED]> wrote:
>
> > Another option is to use a proxy.
> >
> > On Mon, Sep 17, 2012 at 6:44 PM, Morris Bernstein <
> > [EMAIL PROTECTED]> wrote:
> >
> > > The obvious solution would be running multiple Zookeepers
> hierarchically.
> > > Each server would take, say, 1000 clients.  Each server would be a
> client
> > > of an uber Zookeeper.  It'll increase latency some, but it scales.
> > > On Sep 17, 2012 3:57 PM, "Martin Kou" <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm having a use case where there're potentially millions of client
> > > > processes connected to a Zookeeper cluster, but only a very small
> > > > proportion of them would be actively sending requests or receiving
> any
> > > data
> > > > (say, about 1000 of them every second) at any time. Has anyone tried
> > this
> > > > before?
> > > >
> > > > Best Regards,
> > > > Martin Kou
> > > >
> > >
> >
>