Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - HDFS load balancing for non-local reads


Copy link to this message
-
Re: HDFS load balancing for non-local reads
Suresh Srinivas 2012-01-05, 22:32
Currently it sorts the block locations as:
# local node
# local rack node
# random order of remote nodes

See DatanodeManager#sortLocatedBlock(...) and
NetworkTopology#pseudoSortByDistance(...).

You can play around with other policies by plugging in different
NetworkTopology.

On Thu, Jan 5, 2012 at 1:40 PM, Ben Clay <[EMAIL PROTECTED]> wrote:

> Hi-****
>
> ** **
>
> How does the NameNode handle load balancing of non-local reads with
> multiple block locations when locality is equal?****
>
> ** **
>
> IE, if the client is equidistant (same rack) from 2 DataNodes hosting the
> same block, does the NameNode consider current client count or any other
> load indicators when deciding which DataNode will satisfy the read
> request?  Or, is the client provided a list of all split locations and is
> allowed to make this choice themselves?****
>
> ** **
>
> Thanks!****
>
> ** **
>
> -Ben****
>
> ** **
>