Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> Hadoop Master and Slave Discovery


+
Raja Nagendra Kumar 2011-07-03, 02:11
+
Steve Loughran 2011-07-04, 10:44
Copy link to this message
-
Re: Hadoop Master and Slave Discovery
One reasonable suggestion that I have heard recently was to do like Google
does and put a DNS front end onto Zookeeper.  Machines would need to have
DNS set up properly and a requests for a special ZK based domain would have
to be delegated to the fancy DNS setup, but this would allow all kinds of
host targeted configuration settings to be moved by a very standardized
network protocol.  There are difficulties moving port numbers and all you
really get are hostnames, but it is a nice trick since configuration of
which nameserver to use is a common admin task.

On Mon, Jul 4, 2011 at 3:44 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:

> On 03/07/11 03:11, Raja Nagendra Kumar wrote:
>
>>
>> Hi,
>>
>> Instead of depending on local syncup to configuration files, would it be a
>> nice way to adopt JINI Discovery model, where in masters and slaves can
>> discover each other dynamically through a UDP broadcast/heart beat methods
>>
>
> That assumes that UDP Broadcast is supported through the switches (many
> turn it off as it creates too much traffic), or UDP multicast is supported
> (as an example of an infrastructure that does not, play with EC2)
>
>
>  This would mean, any machine can come up and say I am a slave and
>> automatically discover the master and start supporting the master with in<
>> x seconds.
>>
>>
>
> How will your slave determine that the master that it has bonded to is the
> master that it should bond to and not something malicious within the same
> multicast range? It's possible, but you have generally have to have
> configuration files in the worker nodes.
>
> There's nothing to stop your Configuration Management layer using discovery
> or central config servers (Zookeeper, Anubis, LDAP, DNS, ...), which then
> pushes desired state information to the client nodes. These can deal with
> auth and may support infrastructures that don't support broadcast or
> multicast. Such tooling also gives you the ability to push out host table
> config, JVM options, logging parameters, and bounce worker nodes into new
> states without waiting for them to timeout and try to rediscover new
> masters.
>
+
Steve Loughran 2011-07-05, 09:40
+
Eric Yang 2011-07-05, 22:00
+
Steve Loughran 2011-07-06, 09:52
+
Patrick Hunt 2011-07-06, 16:10
+
Eric Yang 2011-07-06, 16:15
+
Patrick Hunt 2011-07-06, 16:18
+
Eric Yang 2011-07-06, 18:25
+
Allen Wittenauer 2011-07-06, 23:02
+
Eric Yang 2011-07-07, 00:05
+
Allen Wittenauer 2011-07-07, 00:49
+
Eric Yang 2011-07-07, 17:17
+
Warren Turkal 2011-07-07, 18:03
+
Warren Turkal 2011-07-07, 17:56