Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> slow transfer rates w/ HDFS balancer

Copy link to this message
Re: slow transfer rates w/ HDFS balancer

On 10/8/09 12:03 PM, "David Howell" <[EMAIL PROTECTED]> wrote:

> I just added a couple of new node to my cluster today, and ran the
> balancer to get some data onto them. With scp, the transfer rate
> between these machines is in the neighborhood of 50 MB/s... but the
> transfers during the balancing cap out at 1 MB/s.
> Is this to be expected, and if not, does anyone have any ideas of
> where to start looking for a problem?

Balancer was intended to run as a background job so that the grid isn't
limping while the balance is going on.  But this is a tunable: change the
dfs.balance.bandwidthPerSec setting.