Hi All,

I've searched for a while on the possibility to configure multiple masters to govern large number of slaves, but I barely found any except for multi-masters for backup in HBase. Is one master enough to support thousands of slaves? Or in real we just configure separate Hadoop instances in a data center(I mean independent master-slaves sets on thousands of hosts)?  

I really appreciate that someone here could enlighten me on those questions!

Thanks and regards
Chengwei

On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:

> Hi All,
>
> I've searched for a while on the possibility to configure multiple masters to govern large number of slaves, but I barely found any except for multi-masters for backup in HBase. Is one master enough to support thousands of slaves? Or in real we just configure separate Hadoop instances in a data center(I mean independent master-slaves sets on thousands of hosts)?  
>
> I really appreciate that someone here could enlighten me on those questions!

This is question #3 on the Hadoop FAQ.

http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F

Thanks for pointing it out!

Because it did not mentioned how may masters in the practice of 4000 nodes, is there only one master managing 4000 slaves?

Thanks
Chengwei

----- Original Message -----
From: "Allen Wittenauer" <[EMAIL PROTECTED]>
To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>
Sent: Wednesday, October 27, 2010 6:08:10 PM
Subject: Re: Multi-Master Hadoop Configuration
On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:

> Hi All,
>
> I've searched for a while on the possibility to configure multiple masters to govern large number of slaves, but I barely found any except for multi-masters for backup in HBase. Is one master enough to support thousands of slaves? Or in real we just configure separate Hadoop instances in a data center(I mean independent master-slaves sets on thousands of hosts)?  
>
> I really appreciate that someone here could enlighten me on those questions!

This is question #3 on the Hadoop FAQ.

http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F

On Oct 27, 2010, at 3:28 PM, Wang, Chengwei wrote:

> Thanks for pointing it out!
>
> Because it did not mentioned how may masters in the practice of 4000 nodes, is there only one master managing 4000 slaves?

Hadoop only supports one master, so yes.

[At least until the AvatarNode gets checked into trunk... altho it might be there now.]

Yes. Only one master, called name-node, is managing HDFS, and only one
master, called job tracker, is managing MapReduce cluster.
You can also read some online documentation, may be even publications.
Thanks,
--Konstantin

On Wed, Oct 27, 2010 at 3:28 PM, Wang, Chengwei <[EMAIL PROTECTED]> wrote:

> Thanks for pointing it out!
>
> Because it did not mentioned how may masters in the practice of 4000 nodes,
> is there only one master managing 4000 slaves?
>
> Thanks
> Chengwei
>
> ----- Original Message -----
> From: "Allen Wittenauer" <[EMAIL PROTECTED]>
> To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>
> Sent: Wednesday, October 27, 2010 6:08:10 PM
> Subject: Re: Multi-Master Hadoop Configuration
>
>
> On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:
>
> > Hi All,
> >
> > I've searched for a while on the possibility to configure multiple
> masters to govern large number of slaves, but I barely found any except for
> multi-masters for backup in HBase. Is one master enough to support thousands
> of slaves? Or in real we just configure separate Hadoop instances in a data
> center(I mean independent master-slaves sets on thousands of hosts)?
> >
> > I really appreciate that someone here could enlighten me on those
> questions!
>
> This is question #3 on the Hadoop FAQ.
>
> http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F
>

Thanks a bunch!

----- Original Message -----
From: "Konstantin Shvachko" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Thursday, October 28, 2010 5:09:27 PM
Subject: Re: Multi-Master Hadoop Configuration

Yes. Only one master, called name-node, is managing HDFS, and only one
master, called job tracker, is managing MapReduce cluster.
You can also read some online documentation, may be even publications.
Thanks,
--Konstantin

On Wed, Oct 27, 2010 at 3:28 PM, Wang, Chengwei <[EMAIL PROTECTED]> wrote:

> Thanks for pointing it out!
>
> Because it did not mentioned how may masters in the practice of 4000 nodes,
> is there only one master managing 4000 slaves?
>
> Thanks
> Chengwei
>
> ----- Original Message -----
> From: "Allen Wittenauer" <[EMAIL PROTECTED]>
> To: "<[EMAIL PROTECTED]>" <[EMAIL PROTECTED]>
> Sent: Wednesday, October 27, 2010 6:08:10 PM
> Subject: Re: Multi-Master Hadoop Configuration
>
>
> On Oct 27, 2010, at 2:55 PM, Wang, Chengwei wrote:
>
> > Hi All,
> >
> > I've searched for a while on the possibility to configure multiple
> masters to govern large number of slaves, but I barely found any except for
> multi-masters for backup in HBase. Is one master enough to support thousands
> of slaves? Or in real we just configure separate Hadoop instances in a data
> center(I mean independent master-slaves sets on thousands of hosts)?
> >
> > I really appreciate that someone here could enlighten me on those
> questions!
>
> This is question #3 on the Hadoop FAQ.
>
> http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F
>

On 27/10/10 23:28, Wang, Chengwei wrote:
> Thanks for pointing it out!
>
> Because it did not mentioned how may masters in the practice of 4000 nodes, is there only one master managing 4000 slaves?
>

Given that you can now get 24 TB worth of HDD in a single server, I
wouldn't worry that much about running out of worker nodes unless you
are feeling very ambitious, as you will have to be well funded to hit
the node limits, and you will probably hit other namenode scalability
limits first, mainly those related to the number of files.