Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> HDFS load balancing for non-local reads


Copy link to this message
-
Re: HDFS load balancing for non-local reads
Currently it sorts the block locations as:
# local node
# local rack node
# random order of remote nodes

See DatanodeManager#sortLocatedBlock(...) and
NetworkTopology#pseudoSortByDistance(...).

You can play around with other policies by plugging in different
NetworkTopology.

On Thu, Jan 5, 2012 at 1:40 PM, Ben Clay <[EMAIL PROTECTED]> wrote:

> Hi-****
>
> ** **
>
> How does the NameNode handle load balancing of non-local reads with
> multiple block locations when locality is equal?****
>
> ** **
>
> IE, if the client is equidistant (same rack) from 2 DataNodes hosting the
> same block, does the NameNode consider current client count or any other
> load indicators when deciding which DataNode will satisfy the read
> request?  Or, is the client provided a list of all split locations and is
> allowed to make this choice themselves?****
>
> ** **
>
> Thanks!****
>
> ** **
>
> -Ben****
>
> ** **
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB