Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> configure hadoop-0.22 fairscheduler


Copy link to this message
-
configure hadoop-0.22 fairscheduler
I want to test version hadoop-0.22.
But when configurate the fairescheduler, I have some troublesome. The
fairscheduler is not efficient.
And I have configured this items in the mapred-site.xml, and also I hava
copy the fairscheduler jar file to the $HADOOP_HOME/lib:

  <property>
    <name>mapreduce.jobtracker.taskScheduler</name>
    <value>org.apache.hadoop.mapred.FairScheduler</value>
  </property>

<property>
        <name>mapred.fairscheduler.allocation.file</name>
        <value>conf/pools.xml</value>
</property>
<property>
        <name>mapred.fairscheduler.preemption</name>
        <value>true</value>
</property>
<property>
        <name>mapred.fairscheduler.assignmultiple</name>
        <value>true</value>
</property>
<property>
        <name>mapred.fairscheduler.poolnameproperty</name>
        <value>mapred.queue.name</value>
        <description>job.set("mapred.queue.name",pool); // pool is set to
either 'high' or 'low' </description>
</property>
<property>
        <name>mapred.queue.names</name>
        <value>default,aaa,bbb</value>
</property>

And the pools.xml in $HADOOP_HOME/conf 's content:

<?xml version="1.0"?>
<allocations>
  <pool name="putindb">
   <minMaps>72</minMaps>
   <minReduces>16</minReduces>
   <maxRunningJobs>20</maxRunningJobs>
   <weight>3.0</weight>
   <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
  </pool>

  <pool name="machinelearning">
   <minMaps>9</minMaps>
   <minReduces>2</minReduces>
   <maxRunningJobs>10</maxRunningJobs>
   <weight>2.0</weight>
   <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
  </pool>

  <pool name="default">
   <minMaps>9</minMaps>
   <minReduces>2</minReduces>
   <maxRunningJobs>10</maxRunningJobs>
   <weight>1.0</weight>
   <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
  </pool>

  <defaultMinSharePreemptionTimeout>60</defaultMinSharePreemptionTimeout>
  <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout>
</allocations>

Can someone help me?
专注于Mysql,MSSQL,Oracle,Hadoop
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB