Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> max 1 mapper per node


Copy link to this message
-
Re: max 1 mapper per node

> We've been against these 'features' since it leads to very bad
> behaviour across the cluster with multiple apps/users etc.
Its not new feature, its extension of existing resource scheduling which
works good enough only for RAM. There are 2 other resources - CPU cores
and network IO which needs to be considered.

We have job which is doing lot of network IO in mapper and its desirable
to run mappers on different nodes even if reading blocks from HDFS will
not be local.

Our second job is burning all CPU cores on machine while doing
computations, its important for mappers not to land on same node.