Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Fair Scheduler of Hadoop


Copy link to this message
-
Fair Scheduler of Hadoop
Hi guys,

I have a quick question regarding to fire scheduler of Hadoop, I am reading
this article =>
http://blog.cloudera.com/blog/2008/11/job-scheduling-in-hadoop/, my
question is from the following statements, "There is currently no support
for preemption of long tasks, but this is being added in
HADOOP-4665<https://issues.apache.org/jira/browse/HADOOP-4665>,
which will allow you to set how long each pool will wait before preempting
other jobs’ tasks to reach its guaranteed capacity.".

My questions are,

1. What means "preemption of long tasks"? Kill long running tasks, or pause
long running tasks to give resources to other tasks, or it means something
else?
2. I am also confused about "set how long each pool will wait before
preempting other jobs’ tasks to reach its guaranteed capacity"., what means
"reach its guaranteed capacity"? I think when using fair scheduler, each
pool has predefined resources allocation settings (and the settings
guarantees each pool has resources as configured), is that true? In what
situations each pool will not have its guaranteed (or configured) capacity?

regards,
Lin
+
Joep Rottinghuis 2013-01-21, 00:24
+
Lin Ma 2013-01-21, 17:30
+
Joep Rottinghuis 2013-01-21, 18:00
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB