Is the number of nodes to be decommissioned bounded by replication factor?

E.g. with a replication factor of 2, is it safe to decommission 2 data nodes at a time?

I guess probably it is fine if namenode simply reads a list of blocks on each node being decommissioned and replicate those blocks to other nodes. But not sure.

Thanks,

Michael
      

On Aug 31, 2010, at 12:58 PM, jiang licht wrote:

> Is the number of nodes to be decommissioned bounded by replication factor?

No, it is bounded mainly by network bandwidth and NN/DN RPC call rate.   I've seen decommissions of like 400 nodes at once.  

Not sure if it's up to date, but the FAQ attempts to answer this question:
http://wiki.apache.org/hadoop/FAQ#A17.

On Tue, Aug 31, 2010 at 1:04 PM, Allen Wittenauer
<[EMAIL PROTECTED]>wrote:

>
> On Aug 31, 2010, at 12:58 PM, jiang licht wrote:
>
> > Is the number of nodes to be decommissioned bounded by replication
> factor?
>
> No, it is bounded mainly by network bandwidth and NN/DN RPC call rate.
> I've seen decommissions of like 400 nodes at once.
>
>

On 01/09/10 08:35, Jeff Hammerbacher wrote:
> Not sure if it's up to date, but the FAQ attempts to answer this question:
> http://wiki.apache.org/hadoop/FAQ#A17.
>
> On Tue, Aug 31, 2010 at 1:04 PM, Allen Wittenauer
> <[EMAIL PROTECTED]>wrote:
>
>>
>> On Aug 31, 2010, at 12:58 PM, jiang licht wrote:
>>
>>> Is the number of nodes to be decommissioned bounded by replication
>> factor?
>>
>> No, it is bounded mainly by network bandwidth and NN/DN RPC call rate.
>> I've seen decommissions of like 400 nodes at once.
>>
>>
>

I've been worrying a bit about network partition events; seems to me
that if a switch fails a large cluster could get overloaded by excessive
failing heartbeats, replication work and incoming client calls.

Some options to mitigate this (HDFS issues, obviously)

-have a way to temporarily say "It's OK to underreplicate anything that
was on these servers" with a list of servers; pull back the replication
count for a while.

-put the namenode into safe mode if a sufficiently large #of workers go
offline within a specific time period