Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Zookeeper >> mail # user >> High availability backend services via zookeeper or TCP load balancer


+
howard chen 2013-02-26, 17:36
+
Jordan Zimmerman 2013-02-26, 18:27
+
Camille Fournier 2013-02-26, 18:35
+
howard chen 2013-02-27, 09:43
Copy link to this message
-
Re: High availability backend services via zookeeper or TCP load balancer
Hi Howard,

There is Apache Helix which provides you higher level primitives like node,
partition, replica and allows you to specify a custom state machine.
Discovery comes automatically with it.

http://helix.incubator.apache.org/

It hides ZK completely from you. And you only need to handle the state
transition rest of the requirement is specification to the system. Also it
has the features like health check where each node can periodically report
the error counts(or any other metrics), and helix can either alert you if
it reaches the threshold or even disable that node.

Helix is basically built as a orchestration engine and it uses zookeeper to
store its  state and there is no SPOF. It also makes it very easy to
operate your cluster and encourages cleaner design in your system.

Encourage you to take a look at it and see what kind of system you are
building, if it is a simple system where all it needs is service discovery
then helix is probably too heavy. But if you envision your system to become
more than what it is today, Helix might help you get there faster.

thanks,
Kishore G
On Wed, Feb 27, 2013 at 1:43 AM, howard chen <[EMAIL PROTECTED]> wrote:

> Hi
>
> On Wed, Feb 27, 2013 at 2:35 AM, Camille Fournier <[EMAIL PROTECTED]
> >wrote:
>
> > You can definitely use ZK for this, as Jordan said. I would really
> question
> > whether writing client-side code to do this vs using something that is
> > really designed for writing load balancers (like haproxy) wouldn't be a
> > better way to do it however. It doesn't sound like you are creating
> > long-lived connections between these clients and services, and instead
> just
> > want to send a request to an ip address that corresponds to the LB for
> that
> > request. Your client-side code is probably going to be buggier and the
> > setup/maintenance more complex than if you use a simple load balancer. If
> > you're already using ZK for a lot of other things and it is really baked
> in
> > to all your clients, maybe this is the easiest thing to do, but I
> wouldn't
> > use ZK just for this purpose.
> >
> >
> Yes, our connection is long lived (similar to web socket).
>
> We don't have any ZK deployment as of today, but sound like the ZK's core
> feature such as subscribe to notification is really interesting.
>
> Of course, one of our concern is as ZK is quite low level for this kind of
> usage, i.e. service discovery and fail over, so we would need to modify our
> client to support this workflow (or by integrating framework like Neflix's
> curator)
>
> Are there any popular choice besides the Neflix's curator?
>
> Thanks for all the input
>
+
Ted Dunning 2013-02-26, 20:55
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB