Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> ZooKeeper performance


Copy link to this message
-
Re: ZooKeeper performance
You can do that, but you don't have very many clients so sharding won't take
you very far.

But don't let me rain on your parade.  ZK is awesome and can be pushed to
pretty incredible lengths.

On Mon, Oct 3, 2011 at 11:29 AM, Dima Gutzeit
<[EMAIL PROTECTED]>wrote:

> Multiple zookeeper clusters with data partitioning (sharding) ?
>
> Regards,
> Dima Gutzeit.
>
> Sent from my iPhone
>
> On 3 באוק 2011, at 20:25, Ted Dunning <[EMAIL PROTECTED]> wrote:
>
> > It definitely can be done, but what if your application grows by 10x?
> >
> > On Mon, Oct 3, 2011 at 10:56 AM, Dima Gutzeit
> > <[EMAIL PROTECTED]>wrote:
> >
> >> Very nice analysis at the link, thanks.
> >>
> >> Assuming my nodes are around the same size ~100 bytes, ensemble of 5
> >> servers and 10 clients can easily support the required ~20,000
> >> operations per second, no ?
> >>
> >> I will run a smoke-test to prove it ofcourse ...
> >>
> >> Regards,
> >> Dima Gutzeit.
> >>
> >> Sent from my iPhone
> >>
> >> On 3 באוק 2011, at 19:41, "Fournier, Camille F."
> >> <[EMAIL PROTECTED]> wrote:
> >>
> >>> It's pretty easy to set up a zk-smoketest to simulate what you are
> doing.
> >> We can't answer this question without knowing how big the data you're
> >> writing, etc etc. I would recommend testing it out yourself on realistic
> >> data sizes.
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ServiceLatencyOverview
> >>>
> >>> -----Original Message-----
> >>> From: Dima Gutzeit [mailto:[EMAIL PROTECTED]]
> >>> Sent: Monday, October 03, 2011 1:36 PM
> >>> To: [EMAIL PROTECTED]
> >>> Subject: Re: ZooKeeper performance
> >>>
> >>> Inline.
> >>>
> >>> Regards,
> >>> Dima Gutzeit.
> >>>
> >>> Sent from my iPhone
> >>>
> >>> On 3 באוק 2011, at 19:31, Ted Dunning <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> Questions in-line
> >>>>
> >>>> On Mon, Oct 3, 2011 at 10:19 AM, Dima Gutzeit
> >>>> <[EMAIL PROTECTED]>wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> I have some performance related question.
> >>>>>
> >>>>> I am running a cluster of 5 zookeeper machines, each one has dual
> quad
> >>>>> core Xeon 2.5 Ghz, 8 gb RAM.
> >>>>>
> >>>>> I want achieve the following numbers:
> >>>>>
> >>>>> 10 clients producing in total (at peaks):
> >>>>>
> >>>>
> >>>> Each producing this?
> >>>
> >>> Total.
> >>>>
> >>>>
> >>>>> 3K add nodes, 3K delete nodes and 10K watches. Per second.
> >>>>>
> >>>>
> >>>> What do you mean by 10K watches?  10,000 watch notifications?
> >>>
> >>> Yes.
> >>>>
> >>>> Does those numbers make any sense ?
> >>>>>
> >>>>
> >>>> This is a bit high.  It sounds like you are trying to use ZK as a
> >> message
> >>>> bus rather than a coordination service.  You can do this, but the
> >> throughput
> >>>> you can achieve is limited.  If you want higher throughput, it is
> better
> >> to
> >>>> have ZK coordinate a higher performance messaging system such as
> Kafka.
> >>
>