Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> Using ZK for real-time group membership notification


+
Otis Gospodnetic 2011-03-18, 21:02
+
Benjamin Reed 2011-03-18, 21:59
+
Otis Gospodnetic 2011-03-19, 04:41
Copy link to this message
-
Re: Using ZK for real-time group membership notification
Hi Otis,

Would it be possible to change your app to use producer-consumer queue? That way,
no document will go unprocessed when an instance goes down.

http://zookeeper.apache.org/doc/r3.3.2/zookeeperTutorial.html#sc_producerConsumerQueues

--Michi

On Mar 18, 2011, at 9:41 PM, Otis Gospodnetic wrote:

> Hi,
>
> Thanks Ben.  Let me describe the context a bit more - once you know what I'm
> trying to do you may have suggestions for how to solve the problem with or
> without ZK.
>
> I have a continuous "stream" of documents that I need to process.  The stream is
> pretty fast (don't know the exact number, but it's many docs a second).  Docs
> live only in-memory in that stream and cannot be saved to disk at any point up
> the stream.
>
> My app listens to this stream.  Because of the high document ingestion rate I
> need N instances of my app to listen to this stream.  So all N apps listen and
> they all "get" the same documents, but only 1 app actually processes each
> document -- "if (docID mod N == appID) then process doc" -- the usual consistent
> hashing stuff.  I'd like to be able to add and remove apps dynamically and have
> the remaining apps realize that "N" has changed.  Similarly, when some app
> instance dies and thus "N" changes, I'd like all the remaining instances to know
> about it.
>
> If my apps don't know the correct "N" then 1/Nth of docs will go unprocessed (if
> the app died or was removed) until the remaining apps adjust their local value
> of "N".
>
>> to deal with this applications can use views, which allow  clients to
>> reconcile differences. for example, if two processes communicate  and
>
> Hm, this requires apps to communicate with each other.  If each app was aware of
> other apps, then I could get the membership count directly using that mechanism,
> although I still wouldn't be able to immediately detect when some app died, at
> least I'm not sure how I could do that.
>
>> one has a different list of members than the other then they can  both
>> consult zookeeper to reconcile or use the membership list with  the
>> highest zxid. the other option is to count on eventually  everyone
>> converging.
>
> Right, if I could live with eventually having the right "N", then I could use ZK
> as described on
> http://eng.wealthfront.com/2010/01/actually-implementing-group-management.html
>
> But say I'm OK with "eventually everyone converging".  Can I use ZK then?  And
> if so, how "eventually" is this "eventually"?  That is, if an app dies, how
> quickly can ZK notify all znode watchers that znode change? A few milliseconds
> or more?
>
> In general, how does one deal with situations like the one I described above,
> where each app is responsible for 1/Nth of work and where N can uncontrollably
> and unexpectedly change?
>
> Thanks!
> Otis
>
>
>
>
>
> ----- Original Message ----
>> From: Benjamin Reed <[EMAIL PROTECTED]>
>> To: [EMAIL PROTECTED]
>> Sent: Fri, March 18, 2011 5:59:43 PM
>> Subject: Re: Using ZK for real-time group membership notification
>>
>> in a distributed setting such an answer is impossible. especially
>> given the  theory of relativity and the speed of light. a machine may
>> fail right after  sending a heart beat or another may come online right
>> after sending a report.  even if zookeeper could provide this you would
>> still have thread scheduling  issues on a local machine that means that
>> you are operating on old  information.
>>
>> to deal with this applications can use views, which allow  clients to
>> reconcile differences. for example, if two processes communicate  and
>> one has a different list of members than the other then they can  both
>> consult zookeeper to reconcile or use the membership list with  the
>> highest zxid. the other option is to count on eventually  everyone
>> converging.
>>
>> i would not develop a distributed system with the  assumption that "all
>> group members know *the exact number of  members at  all times*".
+
Ted Dunning 2011-03-19, 15:57
+
Mathias Herberts 2011-03-19, 06:08
+
Ted Dunning 2011-03-19, 15:56
+
Ted Dunning 2011-03-19, 15:55
+
Fournier, Camille F. [Tec... 2011-03-21, 02:38
+
Ted Dunning 2011-03-18, 22:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB