Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Using ZK for real-time group membership notification


+
Otis Gospodnetic 2011-03-18, 21:02
+
Benjamin Reed 2011-03-18, 21:59
Copy link to this message
-
Re: Using ZK for real-time group membership notification
Hi,

Thanks Ben.  Let me describe the context a bit more - once you know what I'm
trying to do you may have suggestions for how to solve the problem with or
without ZK.

I have a continuous "stream" of documents that I need to process.  The stream is
pretty fast (don't know the exact number, but it's many docs a second).  Docs
live only in-memory in that stream and cannot be saved to disk at any point up
the stream.

My app listens to this stream.  Because of the high document ingestion rate I
need N instances of my app to listen to this stream.  So all N apps listen and
they all "get" the same documents, but only 1 app actually processes each
document -- "if (docID mod N == appID) then process doc" -- the usual consistent
hashing stuff.  I'd like to be able to add and remove apps dynamically and have
the remaining apps realize that "N" has changed.  Similarly, when some app
instance dies and thus "N" changes, I'd like all the remaining instances to know
about it.

If my apps don't know the correct "N" then 1/Nth of docs will go unprocessed (if
the app died or was removed) until the remaining apps adjust their local value
of "N".

> to deal with this applications can use views, which allow  clients to
> reconcile differences. for example, if two processes communicate  and

Hm, this requires apps to communicate with each other.  If each app was aware of
other apps, then I could get the membership count directly using that mechanism,
although I still wouldn't be able to immediately detect when some app died, at
least I'm not sure how I could do that.

> one has a different list of members than the other then they can  both
> consult zookeeper to reconcile or use the membership list with  the
> highest zxid. the other option is to count on eventually  everyone
> converging.

Right, if I could live with eventually having the right "N", then I could use ZK
as described on
http://eng.wealthfront.com/2010/01/actually-implementing-group-management.html

But say I'm OK with "eventually everyone converging".  Can I use ZK then?  And
if so, how "eventually" is this "eventually"?  That is, if an app dies, how
quickly can ZK notify all znode watchers that znode change? A few milliseconds
or more?

In general, how does one deal with situations like the one I described above,
where each app is responsible for 1/Nth of work and where N can uncontrollably
and unexpectedly change?

Thanks!
Otis

----- Original Message ----
> From: Benjamin Reed <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Fri, March 18, 2011 5:59:43 PM
> Subject: Re: Using ZK for real-time group membership notification
>
> in a distributed setting such an answer is impossible. especially
> given the  theory of relativity and the speed of light. a machine may
> fail right after  sending a heart beat or another may come online right
> after sending a report.  even if zookeeper could provide this you would
> still have thread scheduling  issues on a local machine that means that
> you are operating on old  information.
>
> to deal with this applications can use views, which allow  clients to
> reconcile differences. for example, if two processes communicate  and
> one has a different list of members than the other then they can  both
> consult zookeeper to reconcile or use the membership list with  the
> highest zxid. the other option is to count on eventually  everyone
> converging.
>
> i would not develop a distributed system with the  assumption that "all
> group members know *the exact number of  members at  all times*".
>
> ben
>
> On Fri, Mar 18, 2011 at 2:02 PM, Otis  Gospodnetic
> <[EMAIL PROTECTED]>  wrote:
> > Hi,
> >
> > Short version:
> > How can ZK be used to  make sure that all group members know *the exact
>number of
> > members at  all times*?
> >
> > I have an app that can be run on 1 or more servers.   New instances of the
>app
> > come and go, may die, etc. -- the number of  the app instances is completely
(the
+
Michi Mutsuzaki 2011-03-19, 05:05
+
Ted Dunning 2011-03-19, 15:57
+
Mathias Herberts 2011-03-19, 06:08
+
Ted Dunning 2011-03-19, 15:56
+
Ted Dunning 2011-03-19, 15:55
+
Fournier, Camille F. [Tec... 2011-03-21, 02:38
+
Ted Dunning 2011-03-18, 22:51