Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Hadoop Master and Slave Discovery


Copy link to this message
-
Re: Hadoop Master and Slave Discovery
There's a long standing "ZooKeeper DNS server" jira which can be found
here, someone has already created a basic implementation:
https://issues.apache.org/jira/browse/ZOOKEEPER-703

Patrick

On Wed, Jul 6, 2011 at 2:52 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:
> On 05/07/11 23:00, Eric Yang wrote:
>>
>> In another project, I have implemented a bonjour beacon (jmdns) which
>> sit on the Zookeeper nodes to advertise the location of zookeeper
>> servers.  When clients start up, it will discover location of
>> zookeeper through multicast dns.  Once, the server locations are
>> resolved (ip:port and TXT records), the clients shutdown mdns
>> resolvers.  Client proceed to use the resolved list for zookeeper
>> access.  There does not seem to be cpu overhead incurred by the
>> beacon, nor the clients.  If a client could not connect to zookeeper
>> anymore, then it will start mdns resolvers to look for new list of
>> zookeeper servers.  The code for the project is located at:
>>
>> http://github.com/macroadster/hms
>>
>> It may be possible to use similar approach for location resolution,
>> and load rest of the config through zookeeper.
>>
>> regards,
>> Eric
>>
>
> That's interesting; I think it's more important for clients to be able to
> bind dynamically than it is for the cluster machines, as they should be
> managed anyway.
>
> When I was doing the hadoop-in-VM clustering stuff, I had a well-known URL
> to serve up the relevant XML file for the cluster from the JT -all it did
> was relay the request to the JT at whatever host it had been assigned. All
> the clients needed to know was the URL of the config server, and they could
> bootstrap to working against clusters whose FS and JT URLs were different
> from run to run.
>
> zookeeper discovery would benefit a lot of projects
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB