Team,

I have  rack awareness configured and seems to work fine.  My default rep count is 2.  Now I lost one rack  due to switch failure. Here is what I observe

HDFS  continues to write in the existing available rack. It still keeps two copies of each block, but now these blocks are being stored in the same rack.

My questions:

Is this the default HDFS behavior ?
What happens when the failed rack in back online ? Will HDFS automatically rewrite blocks to the other rack ?
Or - Do I have to run the rebalance to make that happen ?

Thanks in advance.

Prem,

Inline.

On Wed, Feb 8, 2012 at 12:58 AM, Jain, Prem <[EMAIL PROTECTED]> wrote:
> Team,
>
> I have  rack awareness configured and seems to work fine.  My default rep
> count is 2.  Now I lost one rack  due to switch failure. Here is what I
> observe
>
> HDFS  continues to write in the existing available rack. It still keeps two
> copies of each block, but now these blocks are being stored in the same
> rack.
>
> My questions:
>
> Is this the default HDFS behavior ?

Yes. This is due to the default block placement policy.

> What happens when the failed rack in back online ? Will HDFS automatically
> rewrite blocks to the other rack ?

No, would not do it automatically.

> Or – Do I have to run the rebalance to make that happen ?

Yes balancer may help. You'll also sometimes have to manually
re-enforce the block placement policy in the stable releases
presently, the policy violation recovery is not automatic:

hadoop fs -setrep -R 3 /
hadoop fs -setrep -R 2 /

HTH.

--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

On 02/07/2012 09:45 PM, Harsh J wrote:
> Yes balancer may help. You'll also sometimes have to manually
> re-enforce the block placement policy in the stable releases
> presently, the policy violation recovery is not automatic:
>
> hadoop fs -setrep -R 3 /
> hadoop fs -setrep -R 2 /
When I execute the first command it goes well, but it halts on executing
the second one. I don't know the reason, but the replication factor
becomes 2 on all datanodes. Is it natural?

--
Mohamed Elsayed

> I have  rack awareness configured and seems to work fine.  My default rep
> count is 2.  Now I lost one rack  due to switch failure. Here is what I
> observe
>
> HDFS  continues to write in the existing available rack. It still keeps
two
> copies of each block, but now these blocks are being stored in the same
> rack.
>
> My questions:
>
> Is this the default HDFS behavior ?

Below is from the 'Hadoop : The Definitive Guide'. So, with a replication
of 2 or 3, the first and the second blocks should be placed on different
racks. Not sure why they are getting into the same rack. It makes sense to
put on a block with replication of 2 on 2 different racks (if available)
considering availability.

**** Hadoop’s default strategy is to place the first replica on the same
node as the client (for clients running outside the cluster, a node is
chosen at random, although the system tries not to pick nodes that are too
full or too busy). The second replica is placed on a different rack from
the first (off-rack), chosen at random. The third replica is placed on the
same rack as the second, but on a different node chosen at random.

Praveen

On Wed, Feb 8, 2012 at 2:47 PM, Mohamed Elsayed <
[EMAIL PROTECTED]> wrote:

> On 02/07/2012 09:45 PM, Harsh J wrote:
>
>> Yes balancer may help. You'll also sometimes have to manually
>> re-enforce the block placement policy in the stable releases
>> presently, the policy violation recovery is not automatic:
>>
>> hadoop fs -setrep -R 3 /
>> hadoop fs -setrep -R 2 /
>>
> When I execute the first command it goes well, but it halts on executing
> the second one. I don't know the reason, but the replication factor becomes
> 2 on all datanodes. Is it natural?
>
> --
> Mohamed Elsayed
>
>

Praveen,

On Sat, Feb 11, 2012 at 9:09 AM, Praveen Sripati
<[EMAIL PROTECTED]> wrote:
> Below is from the 'Hadoop : The Definitive Guide'. So, with a replication of
> 2 or 3, the first and the second blocks should be placed on different racks.
> Not sure why they are getting into the same rack.

Its cause one of his racks fell off the network and he is left with no
racks but one. The writes can obviously not happen to two racks now.

--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about