Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> configure hadoop-0.22 fairscheduler


+
Jameson Li 2012-09-06, 09:17
+
Harsh J 2012-09-08, 05:08
Copy link to this message
-
Re: configure hadoop-0.22 fairscheduler
Hi Harsh,

Thanks for your reply. And I am sorry for my unclear description.

As I mentioned previous, I think I configured the fairsheduler correctly in
hadoop-0.22.0.

But when I commit lots of the jobs:
   many big jobs (map number and reduce number is bigger than the
map/reduce slot) commit first.
   and many small jobs(just 1-2map/reduce per job) commit later.
And I find in the jobtracker http page that it used
the JobQueueTaskScheduler, and even the http://jobtracker:port/scheduler
page is not found. When the big jobs is running, the small job can't start
before big job' complete.

So I guess hadoop-0.22 do not support fairscheduler or somewhere I
configured wrong.

专注于Mysql,MSSQL,Oracle,Hadoop
2012/9/8 Harsh J <[EMAIL PROTECTED]>

> Hey Jameson,
>
> When calling something inefficient, perhaps also share some details on
> how/why/what? How else would we know what you wish to see and what
> you're seeing instead? :)
>
> On Thu, Sep 6, 2012 at 2:47 PM, Jameson Li <[EMAIL PROTECTED]> wrote:
> > I want to test version hadoop-0.22.
> > But when configurate the fairescheduler, I have some troublesome. The
> > fairscheduler is not efficient.
> > And I have configured this items in the mapred-site.xml, and also I hava
> > copy the fairscheduler jar file to the $HADOOP_HOME/lib:
> >
> >   <property>
> >     <name>mapreduce.jobtracker.taskScheduler</name>
> >     <value>org.apache.hadoop.mapred.FairScheduler</value>
> >   </property>
> >
> > <property>
> >         <name>mapred.fairscheduler.allocation.file</name>
> >         <value>conf/pools.xml</value>
> > </property>
> > <property>
> >         <name>mapred.fairscheduler.preemption</name>
> >         <value>true</value>
> > </property>
> > <property>
> >         <name>mapred.fairscheduler.assignmultiple</name>
> >         <value>true</value>
> > </property>
> > <property>
> >         <name>mapred.fairscheduler.poolnameproperty</name>
> >         <value>mapred.queue.name</value>
> >         <description>job.set("mapred.queue.name",pool); // pool is set
> to
> > either 'high' or 'low' </description>
> > </property>
> > <property>
> >         <name>mapred.queue.names</name>
> >         <value>default,aaa,bbb</value>
> > </property>
> >
> > And the pools.xml in $HADOOP_HOME/conf 's content:
> >
> > <?xml version="1.0"?>
> > <allocations>
> >   <pool name="putindb">
> >    <minMaps>72</minMaps>
> >    <minReduces>16</minReduces>
> >    <maxRunningJobs>20</maxRunningJobs>
> >    <weight>3.0</weight>
> >    <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
> >   </pool>
> >
> >   <pool name="machinelearning">
> >    <minMaps>9</minMaps>
> >    <minReduces>2</minReduces>
> >    <maxRunningJobs>10</maxRunningJobs>
> >    <weight>2.0</weight>
> >    <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
> >   </pool>
> >
> >   <pool name="default">
> >    <minMaps>9</minMaps>
> >    <minReduces>2</minReduces>
> >    <maxRunningJobs>10</maxRunningJobs>
> >    <weight>1.0</weight>
> >    <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
> >   </pool>
> >
> >   <defaultMinSharePreemptionTimeout>60</defaultMinSharePreemptionTimeout>
> >   <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout>
> > </allocations>
> >
> > Can someone help me?
> >
> >
> > 专注于Mysql,MSSQL,Oracle,Hadoop
>
>
>
> --
> Harsh J
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB