Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - slow transfer rates w/ HDFS balancer


+
David Howell 2009-10-08, 19:03
Copy link to this message
-
Re: slow transfer rates w/ HDFS balancer
Allen Wittenauer 2009-10-08, 19:06

On 10/8/09 12:03 PM, "David Howell" <[EMAIL PROTECTED]> wrote:

> I just added a couple of new node to my cluster today, and ran the
> balancer to get some data onto them. With scp, the transfer rate
> between these machines is in the neighborhood of 50 MB/s... but the
> transfers during the balancing cap out at 1 MB/s.
>
> Is this to be expected, and if not, does anyone have any ideas of
> where to start looking for a problem?

Balancer was intended to run as a background job so that the grid isn't
limping while the balance is going on.  But this is a tunable: change the
dfs.balance.bandwidthPerSec setting.
+
David Howell 2009-10-08, 19:21