Do the jobs run on the whole cluster or a single rack?
If you write from a single rack, you will get something similar to what you
described, because the default policy is to put one block locally and 2
blocks on the same remote rack. It does check that there is enough place
available, but does not try to balance.
On Thu, Aug 22, 2013 at 9:41 AM, Marc Sturlese <[EMAIL PROTECTED]>wrote:
> Hey there,
> I've set up rack awareness on my hadoop cluster with replication 3. I have
> racks and each contains 50% of the nodes.
> I can see that the blocks are spread on the 2 racks, the problem is that
> nodes from a rack are storing 2 replicas and the nodes of the other rack
> just one. If I launch the hadoop balancer script, it will properly spread
> the replicas across the 2 racks, leaving all nodes with exactly the same
> available disk space but, after jobs are running for hours, the data will
> unbalanced again (rack1 having all nodes with less empty disk space than
> nodes from rack2)
> Any clue whats going on?
> Thanks in advance
> View this message in context:
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.