Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Zookeeper >> mail # user >> ExecutorService over a zookeeper


Copy link to this message
-
Re: ExecutorService over a zookeeper
Віталій Тимчишин:
> Hello.
>
> I am trying to implement distributed executor service over a zookeeper and
> encountered a problem I am not sure how to deal with.
> <SNIP />
Hi,

I've the impression, you're doing it too complicate. Why not have a znode
jobs/ and put all your jobs as persistent znodes in there.
So your job_queue could look like:

jobs/
  job1
  job2
  job3

A consumer can then lock a job by putting an ephemeral node in one of the job
znodes:

jobs/
  job1
  job2/
    .lock
  job3

Other consumers see that somebody is already working on job2. If the consumer
should die, the ephemeral lock will vanish and another consumer could take
over.
You should not publish results in ZK but use some other communication method.
ZK is really not meant as a messaging solution.
Maybe you'd be better served with a standard message queue (e.g. Apache
activeMQ, rabbitMQ) and use ZK only for coordination? If you have really high
message load, you could have a look at Hedwig (google://zookeeper hedwig).

Best regards,

Thomas Koch, http://www.koch.ro