Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Hadoop Master and Slave Discovery


Copy link to this message
-
Re: Hadoop Master and Slave Discovery
One reasonable suggestion that I have heard recently was to do like Google
does and put a DNS front end onto Zookeeper.  Machines would need to have
DNS set up properly and a requests for a special ZK based domain would have
to be delegated to the fancy DNS setup, but this would allow all kinds of
host targeted configuration settings to be moved by a very standardized
network protocol.  There are difficulties moving port numbers and all you
really get are hostnames, but it is a nice trick since configuration of
which nameserver to use is a common admin task.

On Mon, Jul 4, 2011 at 3:44 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:

> On 03/07/11 03:11, Raja Nagendra Kumar wrote:
>
>>
>> Hi,
>>
>> Instead of depending on local syncup to configuration files, would it be a
>> nice way to adopt JINI Discovery model, where in masters and slaves can
>> discover each other dynamically through a UDP broadcast/heart beat methods
>>
>
> That assumes that UDP Broadcast is supported through the switches (many
> turn it off as it creates too much traffic), or UDP multicast is supported
> (as an example of an infrastructure that does not, play with EC2)
>
>
>  This would mean, any machine can come up and say I am a slave and
>> automatically discover the master and start supporting the master with in<
>> x seconds.
>>
>>
>
> How will your slave determine that the master that it has bonded to is the
> master that it should bond to and not something malicious within the same
> multicast range? It's possible, but you have generally have to have
> configuration files in the worker nodes.
>
> There's nothing to stop your Configuration Management layer using discovery
> or central config servers (Zookeeper, Anubis, LDAP, DNS, ...), which then
> pushes desired state information to the client nodes. These can deal with
> auth and may support infrastructures that don't support broadcast or
> multicast. Such tooling also gives you the ability to push out host table
> config, JVM options, logging parameters, and bounce worker nodes into new
> states without waiting for them to timeout and try to rediscover new
> masters.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB