Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Hadoop Master and Slave Discovery


Copy link to this message
-
Re: Hadoop Master and Slave Discovery

On Jul 5, 2011, at 2:40 AM, Steve Loughran wrote:
> 1. you could use DNS proper, by way of Bonjour/avahi. You don't need to be running any mDNS server to support .local, and I would strongly advise against it in a large cluster (because .local resolution puts a lot of CPU load on every server in the subnet).

+1 mDNS doesn't scale to large sizes.  The only number I've ever heard is up to 1000 hosts (not services!) before the whole system falls apart. I don't think it was meant to scale past like a class C subnet.  

Something else to keep in mind:  a lot of network gear gets multicast really really wrong.  There are reasons why network admins are very happy that Hadoop doesn't use multicast.

... and all that's before one talks about the security implications.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB