Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> ExecutorService over a zookeeper

Copy link to this message
Re: ExecutorService over a zookeeper
Віталій Тимчишин:
> Hello.
> I am trying to implement distributed executor service over a zookeeper and
> encountered a problem I am not sure how to deal with.
> <SNIP />

I've the impression, you're doing it too complicate. Why not have a znode
jobs/ and put all your jobs as persistent znodes in there.
So your job_queue could look like:


A consumer can then lock a job by putting an ephemeral node in one of the job


Other consumers see that somebody is already working on job2. If the consumer
should die, the ephemeral lock will vanish and another consumer could take
You should not publish results in ZK but use some other communication method.
ZK is really not meant as a messaging solution.
Maybe you'd be better served with a standard message queue (e.g. Apache
activeMQ, rabbitMQ) and use ZK only for coordination? If you have really high
message load, you could have a look at Hedwig (google://zookeeper hedwig).

Best regards,

Thomas Koch, http://www.koch.ro