Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # dev - which part of Hadoop is responsible of distributing the input file fragments to datanodes?


+
salmakhalil 2012-11-11, 11:06
+
Harsh J 2012-11-11, 11:53
+
salmakhalil 2012-11-14, 17:40
Copy link to this message
-
Re: which part of Hadoop is responsible of distributing the input file fragments to datanodes?
Yanbo Liang 2012-11-15, 08:13
I guess you means to set your own strategy of block distribution.
If this, just hack the code as following clue:
FSNamesystem.getAdditionalBlock() ---> BlockManager.chooseTarget()
 ---> BlockPlacementPolicy.chooseTarget().
And you need to implement your own BlockPlacementPolicy.
Then if the client request addBlock RPC, the NameNode will assign DataNode
to store the replicas as your rules.

2012/11/15 salmakhalil <[EMAIL PROTECTED]>

> What I want to do exactly is redistributing the input file fragments over
> the
> nodes of cluster according some calculations. I need to find the part that
> starts to distribute the input file to add my code instead of.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/which-part-of-Hadoop-is-responsible-of-distributing-the-input-file-fragments-to-datanodes-tp4019530p4020330.html
> Sent from the Hadoop lucene-dev mailing list archive at Nabble.com.
>
+
salmakhalil 2012-11-14, 17:36