Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> slow transfer rates w/ HDFS balancer


+
David Howell 2009-10-08, 19:03
Copy link to this message
-
Re: slow transfer rates w/ HDFS balancer

On 10/8/09 12:03 PM, "David Howell" <[EMAIL PROTECTED]> wrote:

> I just added a couple of new node to my cluster today, and ran the
> balancer to get some data onto them. With scp, the transfer rate
> between these machines is in the neighborhood of 50 MB/s... but the
> transfers during the balancing cap out at 1 MB/s.
>
> Is this to be expected, and if not, does anyone have any ideas of
> where to start looking for a problem?

Balancer was intended to run as a background job so that the grid isn't
limping while the balance is going on.  But this is a tunable: change the
dfs.balance.bandwidthPerSec setting.
+
David Howell 2009-10-08, 19:21
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB