Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Hadoop Master and Slave Discovery


Copy link to this message
-
Re: Hadoop Master and Slave Discovery
Eric, I'd be happy to work with you to get it committed if you'd like
to take a whack. Would be a great addition to contrib.

Patrick

On Wed, Jul 6, 2011 at 9:15 AM, Eric Yang <[EMAIL PROTECTED]> wrote:
> It would be nicer, if it was written in Java.  I think something wrap
> on top of jmdns would be a better fit for Zookeeper.
>
> regards,
> Eric
>
> On Wed, Jul 6, 2011 at 9:10 AM, Patrick Hunt <[EMAIL PROTECTED]> wrote:
>> There's a long standing "ZooKeeper DNS server" jira which can be found
>> here, someone has already created a basic implementation:
>> https://issues.apache.org/jira/browse/ZOOKEEPER-703
>>
>> Patrick
>>
>> On Wed, Jul 6, 2011 at 2:52 AM, Steve Loughran <[EMAIL PROTECTED]> wrote:
>>> On 05/07/11 23:00, Eric Yang wrote:
>>>>
>>>> In another project, I have implemented a bonjour beacon (jmdns) which
>>>> sit on the Zookeeper nodes to advertise the location of zookeeper
>>>> servers.  When clients start up, it will discover location of
>>>> zookeeper through multicast dns.  Once, the server locations are
>>>> resolved (ip:port and TXT records), the clients shutdown mdns
>>>> resolvers.  Client proceed to use the resolved list for zookeeper
>>>> access.  There does not seem to be cpu overhead incurred by the
>>>> beacon, nor the clients.  If a client could not connect to zookeeper
>>>> anymore, then it will start mdns resolvers to look for new list of
>>>> zookeeper servers.  The code for the project is located at:
>>>>
>>>> http://github.com/macroadster/hms
>>>>
>>>> It may be possible to use similar approach for location resolution,
>>>> and load rest of the config through zookeeper.
>>>>
>>>> regards,
>>>> Eric
>>>>
>>>
>>> That's interesting; I think it's more important for clients to be able to
>>> bind dynamically than it is for the cluster machines, as they should be
>>> managed anyway.
>>>
>>> When I was doing the hadoop-in-VM clustering stuff, I had a well-known URL
>>> to serve up the relevant XML file for the cluster from the JT -all it did
>>> was relay the request to the JT at whatever host it had been assigned. All
>>> the clients needed to know was the URL of the config server, and they could
>>> bootstrap to working against clusters whose FS and JT URLs were different
>>> from run to run.
>>>
>>> zookeeper discovery would benefit a lot of projects
>>>
>>>
>>
>