Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Scheduler Questions


Copy link to this message
-
Re: Scheduler Questions
`yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms` parameter is
defaulted to to 2000ms (DEFAULT_MR_AM_TO_RM_HEARTBEAT_INTERVAL_MS). So, the
AM sends a Heart Beat every 2000ms to the RM. Along with the Heart Beat, it
is also piggy backing the resource request which seems to be an overhead.
The resource request from the AM to the RM should be made as and when
required.

Thanks,
Praveen

On Sat, Nov 26, 2011 at 3:56 AM, Patrick Wendell <[EMAIL PROTECTED]> wrote:

> Hey All,
>
> Two questions about the MR2 scheduler code, for anyone more familiar.
>
> - The return type of allocate() suggests that the AM will get some
> instantaneous allocation of resources. Reading through the two current
> schedulers, however, it looks like containers are only allocated after Node
> heartbeats, and that allocate() will rarely return any useful allocation.
> Is that correct or am I missing something?
>
> - The SchedulerNode class maintains state for both a single "reserved"
> container and several "allocated" containers. As far as I can tell only the
> capacity scheduler uses the former, but it's not clear to me what the
> reserved container is for. Is this to enforce hard limits on allocation
> reserved for particular users?
>
> - Patrick
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB