Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> which part of Hadoop is responsible of distributing the input file fragments to datanodes?


Copy link to this message
-
Re: which part of Hadoop is responsible of distributing the input file fragments to datanodes?
I guess you means to set your own strategy of block distribution.
If this, just hack the code as following clue:
FSNamesystem.getAdditionalBlock() ---> BlockManager.chooseTarget()
 ---> BlockPlacementPolicy.chooseTarget().
And you need to implement your own BlockPlacementPolicy.
Then if the client request addBlock RPC, the NameNode will assign DataNode
to store the replicas as your rules.

2012/11/15 salmakhalil <[EMAIL PROTECTED]>

> What I want to do exactly is redistributing the input file fragments over
> the
> nodes of cluster according some calculations. I need to find the part that
> starts to distribute the input file to add my code instead of.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/which-part-of-Hadoop-is-responsible-of-distributing-the-input-file-fragments-to-datanodes-tp4019530p4020330.html
> Sent from the Hadoop lucene-dev mailing list archive at Nabble.com.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB