Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> rebalancing data on disks?


Copy link to this message
-
Re: rebalancing data on disks?
IIUC you can move the data block files from one disk to another until they are evenly distributed. Give it a shot.

Sent from my iPad

On Oct 31, 2011, at 6:49, Tomasz Chmielewski <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have a HDFS cluster consisting of several hosts.
>
> On each node, I add a new disk when the current capacity is close to full.
>
> Right now, every server has more or less such distribution of data:
>
> /dev/sdf              493G  468G   51M 100% /data1
> /dev/sdg              493G  468G   51M 100% /data2
> /dev/sdh              493G  103G  365G  22% /data3
> /dev/sdi              493G  100G  368G  22% /data4
>
>
> So, /dev/sdf and /dev/sdg almost 100% full, and there is lots of free space on /dev/sdh and /dev/sdi.
>
> Disks which are 100% full don't make monitoring very happy.
>
> Is it possible to rebalance data on the disks on one HDFS server (or, more servers)?
>
>
> "hadoop balancer" will want to rebalance data between the servers, but not between the disks on one server.
>
>
> --
> Tomasz Chmielewski
> http://wpkg.org
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB