Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> Hadoop Master and Slave Discovery


+
Raja Nagendra Kumar 2011-07-03, 02:11
+
Steve Loughran 2011-07-04, 10:44
+
Ted Dunning 2011-07-04, 17:22
+
Steve Loughran 2011-07-05, 09:40
+
Eric Yang 2011-07-05, 22:00
+
Steve Loughran 2011-07-06, 09:52
+
Patrick Hunt 2011-07-06, 16:10
+
Eric Yang 2011-07-06, 16:15
+
Patrick Hunt 2011-07-06, 16:18
+
Eric Yang 2011-07-06, 18:25
Copy link to this message
-
Re: Hadoop Master and Slave Discovery

On Jul 5, 2011, at 2:40 AM, Steve Loughran wrote:
> 1. you could use DNS proper, by way of Bonjour/avahi. You don't need to be running any mDNS server to support .local, and I would strongly advise against it in a large cluster (because .local resolution puts a lot of CPU load on every server in the subnet).

+1 mDNS doesn't scale to large sizes.  The only number I've ever heard is up to 1000 hosts (not services!) before the whole system falls apart. I don't think it was meant to scale past like a class C subnet.  

Something else to keep in mind:  a lot of network gear gets multicast really really wrong.  There are reasons why network admins are very happy that Hadoop doesn't use multicast.

... and all that's before one talks about the security implications.
+
Eric Yang 2011-07-07, 00:05
+
Allen Wittenauer 2011-07-07, 00:49
+
Eric Yang 2011-07-07, 17:17
+
Warren Turkal 2011-07-07, 18:03
+
Warren Turkal 2011-07-07, 17:56
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB