Hadoop, mail # user - Re: HDFS unbalance issue. (HBase over HDFS)

HDFS unbalance issue. (HBase over HDFS)

schubert zhang 2009-03-25, 09:12

Hi all,
I am using hbase-0.19.1 and hadoop-0.19.
My cluster have 5+1 nodes, and there are about 512 regions in HBase (256MB
per region).

But I found the blocks in HDFS is very unbalanced. Following is the status
from HDFS web GUI.

(Node: I don't know if this mailing list can display html!)

HDFS blocks:
node1 509036 blocks
node2 157937 blocks
node3 15783 blocks
node4 15117 blocks
node5 20158 blocks

But my HBase regions are very balanced.
node1 88 regions
node2 108 regions
node3 111 regions
node4 102 regions
node5 105 regions

NodeLast
ContactAdmin StateConfigured
Capacity (GB)Used
(GB)Non DFS
Used (GB)Remaining
(GB)Used
(%)Used
(%)Remaining
(%)Blocksnd1-rack0-cloud<http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F>
1In Service1215.6152.3762.911100.324.3190.5220158
But my HBase regions are very balanced.

AddressStart CodeLoadnd1-rack0-cloud:60020 <http://nd1-rack0-cloud:60030/>
1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/>1237788871362requests=422,
regions=108, usedHeap=1433,
maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/>1237788859541requests=369,
regions=102, usedHeap=1059,
maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
1237788899331requests=384, regions=105, usedHeap=1535,
maxHeap=1983Total:servers:
5 requests=2520, regions=514

Re: HDFS unbalance issue. (HBase over HDFS)

Ryan Rawson 2009-03-25, 09:14

Try
hadoop/bin/start-balancer.sh

HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
whereas balancing in HBase just means opening a file on a different machine.

On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <[EMAIL PROTECTED]> wrote:

> Hi all,
> I am using hbase-0.19.1 and hadoop-0.19.
> My cluster have 5+1 nodes, and there are about 512 regions in HBase (256MB
> per region).
>
> But I found the blocks in HDFS is very unbalanced. Following is the status
> from HDFS web GUI.
>
> (Node: I don't know if this mailing list can display html!)
>
> HDFS blocks:
> node1 509036 blocks
> node2 157937 blocks
> node3 15783 blocks
> node4 15117 blocks
> node5 20158 blocks
>
> But my HBase regions are very balanced.
> node1 88 regions
> node2 108 regions
> node3 111 regions
> node4 102 regions
> node5 105 regions
>
>
>
> NodeLast
> ContactAdmin StateConfigured
> Capacity (GB)Used
> (GB)Non DFS
> Used (GB)Remaining
> (GB)Used
> (%)Used
> (%)Remaining
> (%)Blocksnd1-rack0-cloud<
> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
> >
> 1In Service1215.6152.3762.911100.324.3190.5220158
>
>
> But my HBase regions are very balanced.
>
> AddressStart CodeLoadnd1-rack0-cloud:60020 <http://nd1-rack0-cloud:60030/>
> 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
> nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
> >1237788871362requests=422,
> regions=108, usedHeap=1433,
> maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
> 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
> nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
> >1237788859541requests=369,
> regions=102, usedHeap=1059,
> maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
> 1237788899331requests=384, regions=105, usedHeap=1535,
> maxHeap=1983Total:servers:
> 5 requests=2520, regions=514
>

Re: HDFS unbalance issue. (HBase over HDFS)

schubert zhang 2009-03-25, 10:28

Thanks Ryan. Balancer may take a long time.

The number of block are too different. But maybe it is caused by HBase not
deleting garbage blocks on regionserver1 and regionserver2 and maybe others.

We grep the logs of hadoop and find there is no any "deleting block" in
node1 and node2.

Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
hasoop logs:

namenode:

-----grep -c "ask 10.24.1.12:50010 to delete"-----node1
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
4754
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
1062
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
0

-----grep -c "ask 10.24.1.14:50010 to delete"-----node2
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
1494
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
3305
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
3385
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
1494

-----grep -c "ask 10.24.1.16:50010 to delete"-----node3
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
8022
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
8238
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
4302

-----grep -c "ask 10.24.1.18:50010 to delete"-----node4
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
8591
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
9111
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
5038

-----grep -c "ask 10.24.1.20:50010 to delete"-----node5
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
3794
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
3946
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2989

So, I think it may caused by HBase.
I just grep the log of the zero "delete block" node. and find:
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
104739
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
465927
[schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
hadoop-schubert-datanode-nd1-rack0-cloud.log
0
On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:

> Try
> hadoop/bin/start-balancer.sh
>
> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
> whereas balancing in HBase just means opening a file on a different
> machine.
>
> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <[EMAIL PROTECTED]> wrote:
>
> > Hi all,
> > I am using hbase-0.19.1 and hadoop-0.19.
> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
> (256MB
> > per region).
> >
> > But I found the blocks in HDFS is very unbalanced. Following is the
> status
> > from HDFS web GUI.
> >
> > (Node: I don't know if this mailing list can display html!)
> >
> > HDFS blocks:
> > node1 509036 blocks
> > node2 157937 blocks
> > node3 15783 blocks
> > node4 15117 blocks

Re: HDFS unbalance issue. (HBase over HDFS)

schubert zhang 2009-03-25, 10:37

>From another point of view, I think HBase cannot control to delete blocks on
which node, it would just delete files, and HDFS delete blocks where the
blocks locating.

Schubert

On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang <[EMAIL PROTECTED]> wrote:

> Thanks Ryan. Balancer may take a long time.
>
> The number of block are too different. But maybe it is caused by HBase not
> deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
>
> We grep the logs of hadoop and find there is no any "deleting block" in
> node1 and node2.
>
> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
> hasoop logs:
>
> namenode:
>
> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 4754
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 1062
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 0
>
> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3305
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3385
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
>
> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8022
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 8238
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 4302
>
> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8591
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 9111
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 5038
>
> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3794
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3946
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 2989
>
> So, I think it may caused by HBase.
> I just grep the log of the zero "delete block" node. and find:
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
> 104739
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
> 465927
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log
> 0
>
>
>
>
> On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <[EMAIL PROTECTED]> wrote:
>
>> Try
>> hadoop/bin/start-balancer.sh
>>
>> HDFS doesnt auto-balance. Balancing in HDFS requires moving data around,
>> whereas balancing in HBase just means opening a file on a different
>> machine.
>>
>> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Hi all,

Re: HDFS unbalance issue. (HBase over HDFS)

schubert zhang 2009-03-25, 12:34

Then, I stop my application (the application write to and read from HBase).
After one hour, when I come back to see the status of HDFS, some blocks are
deleted. Following is current status.

[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2956
[schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
hadoop-schubert-namenode-nd0-rack0-cloud.log
2962

node1: 464518
node2: 42495
node3: 7505
node4: 7205
node5: 7636

On each node, the datanode process is busy (top).

I want to know the reason of these phenomenons. Thanks.

Schubert

On Wed, Mar 25, 2009 at 6:37 PM, schubert zhang <[EMAIL PROTECTED]> wrote:

> From another point of view, I think HBase cannot control to delete blocks
> on which node, it would just delete files, and HDFS delete blocks where the
> blocks locating.
>
> Schubert
>
> On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang <[EMAIL PROTECTED]> wrote:
>
>> Thanks Ryan. Balancer may take a long time.
>>
>> The number of block are too different. But maybe it is caused by HBase not
>> deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
>>
>> We grep the logs of hadoop and find there is no any "deleting block" in
>> node1 and node2.
>>
>> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
>> hasoop logs:
>>
>> namenode:
>>
>> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 4754
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 1062
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 0
>>
>> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 1494
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 3305
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 3385
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 1494
>>
>> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 8022
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 8238
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 4302
>>
>> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 8591
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 9111
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 5038
>>
>> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
>> 3794
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
>> 3946
>> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
>> hadoop-schubert-namenode-nd0-rack0-cloud.log
>> 2989
>>
>> So, I think it may caused by HBase.
>> I just grep the log of the zero "delete block" node. and find:

Re: HDFS unbalance issue. (HBase over HDFS)

schubert zhang 2009-03-26, 03:07

Tanks Andrew and Billy.
I think the subject of this mail thread is not appropriate, it may not be a
balance issue.
The problem seems the block deleting scheduler in HDFS.

Last night(timezone:+8), I slow down my application, and this morning, I
found almost all garbage blocks are deleted.
Here is the current blocks number of each datanode:
node1: 10651
node2: 10477
node3: 12185
node4: 11607
node5: 14000

It seems fine.
But I want to study the code of HDFS and make clear the policy of deleting
blocks on datanodes. If anyone in the hadoop community can give me some
advices?

Schubert

On Thu, Mar 26, 2009 at 7:55 AM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
>
> > From: schubert zhang <[EMAIL PROTECTED]>
> > From another point of view, I think HBase cannot control to
> > delete blocks on which node, it would just delete files, and
> > HDFS delete blocks where the blocks locating.
>
> Yes, that is exactly correct.
>
> Best regards,
>
> - Andy
>
>
>
>
>

Re: HDFS unbalance issue. (HBase over HDFS)

schubert zhang 2009-03-27, 07:49

Thanks Samuel,
Your information is very correct.
I have also read code about garbage collection of invalidating blocks.

But I found the namenode is fair to process the invalidating for each
datanode.
In my cluster, there are 5 datanode. The storage IDs are:

node1: DS- 978762906-10.24.1.12-50010-1237686434530
node2: DS- 489086185-10.24.1.14-50010-1237686416330
node3: DS-1170985665-10.24.1.16-50010-1237686426395
node4: DS-1024388083-10.24.1.18-50010-1237686404482
node5: DS-2136798339-10.24.1.20-50010-1237686444430
I know the storage ID is generated
by org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).

In org.apache.hadoop.hdfs.server.namenode.FSNamesystem

  // Keeps a Collection for every named machine containing
  // blocks that have recently been invalidated and are thought to live
  // on the machine in question.
  // Mapping: StorageID -> ArrayList<Block>
  //
  private Map<String, Collection<Block>> recentInvalidateSets new TreeMap<String, Collection<Block>>();

In org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
This thread run in interval: replicationRecheckInterval=3000 milliseconds.

Into computeDatanodeWork()
nodesToProcess = 2.

then into computeInvalidateWork(nodesToProcess)
the for cycle will only exec 2 cycles.

for each cycle, go into invalidateWorkForOneNode()
it will always get the first node to invalidate blocks on this node.
  String firstNodeId = recentInvalidateSets.keySet().iterator().next();

TreeMap is a stored map, so, the ketSet is:
[1024388083-10.24.1.18-50010-1237686404482,
1170985665-10.24.1.16-50010-1237686426395,
2136798339-10.24.1.20-50010-1237686444430,
489086185-10.24.1.14-50010-1237686416330,
978762906-10.24.1.12-50010-1237686434530]

So, the sequence of node list in recentInvalidateSets is:
[node4, node3, node5, node2, node1]

So, every time in invalidateWorkForOneNode(), it will always process node4
then node3, then node2 and then node1.

My application is a HBase write-heavy application.
So, there is many blocks need invalidate in each datanode. So when each 3000
milliseconds, at most, there is only two datanode is processed. Since the
node1 is the last one in the TreeMap, it have no change to be garbage
collected.

I think HDFS namenode should fix this issue.

Schubert
On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <[EMAIL PROTECTED]> wrote:

> After a file is deleted, HDFS does not immediately reclaim the available
> physical storage. It does so only lazily during garbage collection. When a
> file is deleted by the application, the master remove the file's metadata
> from *FSNamesystem* and logs the deletion immediately. And the file's
> deleted blocks information will be collected in each DataNodeDescriptor's
> *invalidateBlocks* set in Namenode. During the heartbeats between NN and
> DN,
> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks set,
> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
> BlockCommand
> to DN. And the *BlockScanner* thread running on DN will scan, find and
> delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
>
> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files,
> and find the logic of the garbage collection. Hope it will be helpful.
>
> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <[EMAIL PROTECTED]>
> wrote:
>
> > Tanks Andrew and Billy.
> > I think the subject of this mail thread is not appropriate, it may not be
> a
> > balance issue.
> > The problem seems the block deleting scheduler in HDFS.
> >
> > Last night(timezone:+8), I slow down my application, and this morning, I
> > found almost all garbage blocks are deleted.
> > Here is the current blocks number of each datanode:
> > node1: 10651
> > node2: 10477
> > node3: 12185
> > node4: 11607
> > node5: 14000
> >
> > It seems fine.
> > But I want to study the code of HDFS and make clear the policy of
> deleting
> > blocks on datanodes. If anyone in the hadoop community can give me some

Re: HDFS unbalance issue. (HBase over HDFS)

schubert zhang 2009-03-27, 07:50

Sorry "But I found the namenode is fair to process the invalidating for each
datanode."
should be:

"But I found the namenode is unfair to process the invalidating for each
datanode."
On Fri, Mar 27, 2009 at 3:49 PM, schubert zhang <[EMAIL PROTECTED]> wrote:

> Thanks Samuel,
> Your information is very correct.
> I have also read code about garbage collection of invalidating blocks.
>
> But I found the namenode is fair to process the invalidating for each
> datanode.
> In my cluster, there are 5 datanode. The storage IDs are:
>
> node1: DS- 978762906-10.24.1.12-50010-1237686434530
> node2: DS- 489086185-10.24.1.14-50010-1237686416330
> node3: DS-1170985665-10.24.1.16-50010-1237686426395
> node4: DS-1024388083-10.24.1.18-50010-1237686404482
> node5: DS-2136798339-10.24.1.20-50010-1237686444430
> I know the storage ID is generated
> by org.apache.hadoop.hdfs.server.datanode.DataNode.setNewStorageID(...).
>
> In org.apache.hadoop.hdfs.server.namenode.FSNamesystem
>
> // Keeps a Collection for every named machine containing
> // blocks that have recently been invalidated and are thought to live
> // on the machine in question.
> // Mapping: StorageID -> ArrayList<Block>
> //
> private Map<String, Collection<Block>> recentInvalidateSets > new TreeMap<String, Collection<Block>>();
>
> In org.apache.hadoop.hdfs.server.namenode.FSNamesystem.ReplicationMonitor
> This thread run in interval: replicationRecheckInterval=3000 milliseconds.
>
> Into computeDatanodeWork()
> nodesToProcess = 2.
>
> then into computeInvalidateWork(nodesToProcess)
> the for cycle will only exec 2 cycles.
>
> for each cycle, go into invalidateWorkForOneNode()
> it will always get the first node to invalidate blocks on this node.
> String firstNodeId = recentInvalidateSets.keySet().iterator().next();
>
> TreeMap is a stored map, so, the ketSet is:
> [1024388083-10.24.1.18-50010-1237686404482,
> 1170985665-10.24.1.16-50010-1237686426395,
> 2136798339-10.24.1.20-50010-1237686444430,
> 489086185-10.24.1.14-50010-1237686416330,
> 978762906-10.24.1.12-50010-1237686434530]
>
> So, the sequence of node list in recentInvalidateSets is:
> [node4, node3, node5, node2, node1]
>
> So, every time in invalidateWorkForOneNode(), it will always process node4
> then node3, then node2 and then node1.
>
> My application is a HBase write-heavy application.
> So, there is many blocks need invalidate in each datanode. So when each
> 3000 milliseconds, at most, there is only two datanode is processed. Since
> the node1 is the last one in the TreeMap, it have no change to be garbage
> collected.
>
> I think HDFS namenode should fix this issue.
>
> Schubert
>
> On Thu, Mar 26, 2009 at 2:57 PM, Samuel Guo <[EMAIL PROTECTED]> wrote:
>
>> After a file is deleted, HDFS does not immediately reclaim the available
>> physical storage. It does so only lazily during garbage collection. When a
>> file is deleted by the application, the master remove the file's metadata
>> from *FSNamesystem* and logs the deletion immediately. And the file's
>> deleted blocks information will be collected in each DataNodeDescriptor's
>> *invalidateBlocks* set in Namenode. During the heartbeats between NN and
>> DN,
>> NN will scan the specified DN's DataNodeDescriptor's invalidateBlocks set,
>> find the blocks to be deleted in DN and send a *DNA_INVALIDATE*
>> BlockCommand
>> to DN. And the *BlockScanner* thread running on DN will scan, find and
>> delete these blocks after DN receives the *DNA_INVALIDATE* BlockCommand.
>>
>> You can search *DNA_INVALIDATE* in DataNode.java and NameNode.java files,
>> and find the logic of the garbage collection. Hope it will be helpful.
>>
>> On Thu, Mar 26, 2009 at 11:07 AM, schubert zhang <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Tanks Andrew and Billy.
>> > I think the subject of this mail thread is not appropriate, it may not
>> be a
>> > balance issue.
>> > The problem seems the block deleting scheduler in HDFS.
>> >
>> > Last night(timezone:+8), I slow down my application, and this morning, I