Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> extremely imbalance in the hdfs cluster


Copy link to this message
-
Re: extremely imbalance in the hdfs cluster
We have run into this issue as well. Since hadoop is RR writing different
size disks really screw things up royally especially if you are running at
high capacity. We have found that decommissioning hosts for stretches of
time is more effective then the balancer in extreme situations. Another
hokey trick is that nodes that launch a job always use that node as the
first replica. You can leverage that by launching jobs from your bigger
machines which makes data more likely to be saved there. Super hokey
solution is moving blocks around with rsync! (block reports later happen and
deal with this (I do not suggest this)).

Hadoop really does need a more intelligent system then Round Robin writing
for heterogeneous systems, there might be a jira open on this somewhere. But
if you are on 0.20.X you have to work with it.

Edward

On Wed, Jun 29, 2011 at 9:06 AM, 茅旭峰 <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm running a 37 DN hdfs cluster. There are 12 nodes have 20TB capacity
> each
> node, and the other 25 nodes have 24TB each node.Unfortunately, there are
> several nodes that contain much more data than others, and I can still see
> the data increasing crazy. The 'dstat' shows
>
> dstat -ta 2
> -----time----- ----total-cpu-usage---- -dsk/total- -net/total- ---paging--
> ---system--
>  date/time   |usr sys idl wai hiq siq| read  writ| recv  send|  in   out |
> int   csw
> 24-06 00:42:43|  1   1  95   2   0   0|  25M   62M|   0     0 |   0   0.1
> |3532  5644
> 24-06 00:42:45|  7   1  91   0   0   0|  16k  176k|8346B 1447k|   0     0
> |1201   365
> 24-06 00:42:47|  7   1  91   0   0   0|  12k  172k|9577B 1493k|   0     0
> |1223   334
> 24-06 00:42:49| 11   3  83   1   0   1|  26M   11M|  78M   66M|   0     0 |
>  12k   18k
> 24-06 00:42:51|  4   3  90   1   0   2|  17M  181M| 117M   53M|   0     0 |
>  15k   26k
> 24-06 00:42:53|  4   3  87   4   0   2|  15M  375M| 117M   55M|   0     0 |
>  16k   26k
> 24-06 00:42:55|  3   2  94   1   0   1|  15M   37M|  80M   17M|   0     0 |
>  10k   15k
> 24-06 00:42:57|  0   0  98   1   0   0|  18M   23M|7259k 5988k|   0     0
> |1932  1066
> 24-06 00:42:59|  0   0  98   1   0   0|  16M  132M| 708k  106k|   0     0
> |1484   491
> 24-06 00:43:01|  4   2  91   2   0   1|  23M   64M|  76M   41M|   0     0
> |8441    13k
> 24-06 00:43:03|  4   3  88   3   0   1|  17M  207M|  91M   48M|   0     0 |
>  11k   16k
>
> From the result of dstat, we can see that the throughput of write is much
> more than read.
> I've started a balancer processor, with dfs.balance.bandwidthPerSec set to
> bytes. From
> the balancer log, I can see the balancer works well. But the balance
> operation can not
> catch up with the write operation.
>
> Now I can only stop the mad increase of data size by stopping the datanode,
> and setting
> dfs.datanode.du.reserved 300GB, then starting the datanode again. Until the
> total size
> reaches the 300GB reservation line, the increase stopped.
>
> The output of 'hadoop dfsadmin -report' shows for the crazy nodes,
>
> Name: 10.150.161.88:50010
> Decommission Status : Normal
> Configured Capacity: 20027709382656 (18.22 TB)
> DFS Used: 14515387866480 (13.2 TB)
> Non DFS Used: 0 (0 KB)
> DFS Remaining: 5512321516176(5.01 TB)
> DFS Used%: 72.48%
> DFS Remaining%: 27.52%
> Last contact: Wed Jun 29 21:03:01 CST 2011
>
>
> Name: 10.150.161.76:50010
> Decommission Status : Normal
> Configured Capacity: 20027709382656 (18.22 TB)
> DFS Used: 16554450730194 (15.06 TB)
> Non DFS Used: 0 (0 KB)
> DFS Remaining: 3473258652462(3.16 TB)
> DFS Used%: 82.66%
> DFS Remaining%: 17.34%
> Last contact: Wed Jun 29 21:03:02 CST 2011
>
> while the other normal datanode, it just like
>
> Name: 10.150.161.65:50010
> Decommission Status : Normal
> Configured Capacity: 23627709382656 (21.49 TB)
> DFS Used: 5953984552236 (5.42 TB)
> Non DFS Used: 1200643810004 (1.09 TB)
> DFS Remaining: 16473081020416(14.98 TB)
> DFS Used%: 25.2%
> DFS Remaining%: 69.72%
> Last contact: Wed Jun 29 21:03:01 CST 2011
>
>
> Name: 10.150.161.80:50010
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB