In my existing 48-node cluster, the engineer who originally designed it (no longer here) did not specify logical racks in the HDFS configuration, instead leaving everything in "default-rack".  Now I have 4 physical racks of machines, and I am becoming concerned about failure and near/far replication issues.

Anyone have any ideas what will happen if I tell hadoop about the physical rack layout (i.e. nuke default-rack, and create rack110, rack111, rack112, rack113, etc)?

-j

On Feb 4, 2011, at 5:00 PM, Jonathan Disher wrote:

> In my existing 48-node cluster, the engineer who originally designed it (no longer here) did not specify logical racks in the HDFS configuration, instead leaving everything in "default-rack".  Now I have 4 physical racks of machines, and I am becoming concerned about failure and near/far replication issues.
>
> Anyone have any ideas what will happen if I tell hadoop about the physical rack layout (i.e. nuke default-rack, and create rack110, rack111, rack112, rack113, etc)?
What will happen is that your next fsck run will be full of replication policy violation errors/warnings (I forget which).  These are non-fatal and trivial to fix if you have enough space:

a) use setrep to increase the replication factor of all files
b) let the namenode re-replicate all blocks
c) use setrep again to decrease the replication factor of all files
d) let the namenode remove an extra replica:  it will almost always choose a replica that violates policy

Is it worth fixing? Yes, definitely, for the reasons you suspect:  if you lose a rack, you might lose data.