Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Different container sizes for MR tasks, job does not complete


Copy link to this message
-
Different container sizes for MR tasks, job does not complete
Hi,

I am using MapReduce on YARN. I want to make tasks of the same job run
in containers with different sizes. For example: Job1 = <Task1, ...,
Task8>. Task1 := 1280 MB; Task2 to 8  := 1024 MB.
To achieve this, I manually call
reqEvent.getCapability().setMemory(MEMORY_SIZE) in
RMContainerAllocator.java with MEMORY_SIZE = 1280 or 1024. The problem
is, if one task has different memory than the others, then the whole job
stops at 80% forever. When all the tasks have the same memory, the job
completes correctly and fast. Does some one has any idea?

Thanks a lot.

Yuzhang
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB