Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper, mail # user - Using ZK for real-time group membership notification


+
Otis Gospodnetic 2011-03-18, 21:02
+
Benjamin Reed 2011-03-18, 21:59
+
Otis Gospodnetic 2011-03-19, 04:41
Copy link to this message
-
Re: Using ZK for real-time group membership notification
Michi Mutsuzaki 2011-03-19, 05:05
Hi Otis,

Would it be possible to change your app to use producer-consumer queue? That way,
no document will go unprocessed when an instance goes down.

http://zookeeper.apache.org/doc/r3.3.2/zookeeperTutorial.html#sc_producerConsumerQueues

--Michi

On Mar 18, 2011, at 9:41 PM, Otis Gospodnetic wrote:

> Hi,
>
> Thanks Ben.  Let me describe the context a bit more - once you know what I'm
> trying to do you may have suggestions for how to solve the problem with or
> without ZK.
>
> I have a continuous "stream" of documents that I need to process.  The stream is
> pretty fast (don't know the exact number, but it's many docs a second).  Docs
> live only in-memory in that stream and cannot be saved to disk at any point up
> the stream.
>
> My app listens to this stream.  Because of the high document ingestion rate I
> need N instances of my app to listen to this stream.  So all N apps listen and
> they all "get" the same documents, but only 1 app actually processes each
> document -- "if (docID mod N == appID) then process doc" -- the usual consistent
> hashing stuff.  I'd like to be able to add and remove apps dynamically and have
> the remaining apps realize that "N" has changed.  Similarly, when some app
> instance dies and thus "N" changes, I'd like all the remaining instances to know
> about it.
>
> If my apps don't know the correct "N" then 1/Nth of docs will go unprocessed (if
> the app died or was removed) until the remaining apps adjust their local value
> of "N".
>
>> to deal with this applications can use views, which allow  clients to
>> reconcile differences. for example, if two processes communicate  and
>
> Hm, this requires apps to communicate with each other.  If each app was aware of
> other apps, then I could get the membership count directly using that mechanism,
> although I still wouldn't be able to immediately detect when some app died, at
> least I'm not sure how I could do that.
>
>> one has a different list of members than the other then they can  both
>> consult zookeeper to reconcile or use the membership list with  the
>> highest zxid. the other option is to count on eventually  everyone
>> converging.
>
> Right, if I could live with eventually having the right "N", then I could use ZK
> as described on
> http://eng.wealthfront.com/2010/01/actually-implementing-group-management.html
>
> But say I'm OK with "eventually everyone converging".  Can I use ZK then?  And
> if so, how "eventually" is this "eventually"?  That is, if an app dies, how
> quickly can ZK notify all znode watchers that znode change? A few milliseconds
> or more?
>
> In general, how does one deal with situations like the one I described above,
> where each app is responsible for 1/Nth of work and where N can uncontrollably
> and unexpectedly change?
>
> Thanks!
> Otis
>
>
>
>
>
> ----- Original Message ----
>> From: Benjamin Reed <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Fri, March 18, 2011 5:59:43 PM
>> Subject: Re: Using ZK for real-time group membership notification
>>
>> in a distributed setting such an answer is impossible. especially
>> given the  theory of relativity and the speed of light. a machine may
>> fail right after  sending a heart beat or another may come online right
>> after sending a report.  even if zookeeper could provide this you would
>> still have thread scheduling  issues on a local machine that means that
>> you are operating on old  information.
>>
>> to deal with this applications can use views, which allow  clients to
>> reconcile differences. for example, if two processes communicate  and
>> one has a different list of members than the other then they can  both
>> consult zookeeper to reconcile or use the membership list with  the
>> highest zxid. the other option is to count on eventually  everyone
>> converging.
>>
>> i would not develop a distributed system with the  assumption that "all
>> group members know *the exact number of  members at  all times*".
+
Ted Dunning 2011-03-19, 15:57
+
Mathias Herberts 2011-03-19, 06:08
+
Ted Dunning 2011-03-19, 15:56
+
Ted Dunning 2011-03-19, 15:55
+
Fournier, Camille F. [Tec... 2011-03-21, 02:38
+
Ted Dunning 2011-03-18, 22:51