Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - DFS rebalance


Copy link to this message
-
Re: DFS rebalance
Harsh J 2012-06-10, 19:05
Hey Eric,

HDFS does not balance imbalanced DNs automatically. However DN
imbalance is not something HDFS causes, its more to do with client
write patterns. Ensure that your HDFS clients write from an edge node,
not a DN node, else one block always gets written locally and that
causes an imbalance.

Running the HDFS balancer on a live HBase cluster is no-problem. Do
not worry, and just do it. You will lose a bit of data locality for
about some time, for some regions, but no destruction of data or loss
of service.

On Tue, Jun 5, 2012 at 11:25 PM, Eric Raymond <[EMAIL PROTECTED]> wrote:
> Hello all,
>
> I am not sure this goes here or not, but I wanted to get a formal answer on
> using the hdfs balancer with hbase.  From what I heard a while back, its
> not recommended to run alongside hbase, as it can destroy the regions.  How
> can I rebalance my datanode/dfs cluster as, it doesnt seem to do this
> automatically?  I have dfs.balance.bandwidthPerSec set to 100Mbit, but I am
> seeing that a majority of the nodes are reaching 99% full, while 3-4 others
> are only at 30%.
>
> Can anyone advise on this?
>
> Thanks,
>
> Eric

--
Harsh J