Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - configure hadoop-0.22 fairscheduler


Copy link to this message
-
Re: configure hadoop-0.22 fairscheduler
Jameson Li 2012-09-11, 08:21
Hi Harsh,

Thanks for your reply. And I am sorry for my unclear description.

As I mentioned previous, I think I configured the fairsheduler correctly in
hadoop-0.22.0.

But when I commit lots of the jobs:
   many big jobs (map number and reduce number is bigger than the
map/reduce slot) commit first.
   and many small jobs(just 1-2map/reduce per job) commit later.
And I find in the jobtracker http page that it used
the JobQueueTaskScheduler, and even the http://jobtracker:port/scheduler
page is not found. When the big jobs is running, the small job can't start
before big job' complete.

So I guess hadoop-0.22 do not support fairscheduler or somewhere I
configured wrong.

专注于Mysql,MSSQL,Oracle,Hadoop
2012/9/8 Harsh J <[EMAIL PROTECTED]>

> Hey Jameson,
>
> When calling something inefficient, perhaps also share some details on
> how/why/what? How else would we know what you wish to see and what
> you're seeing instead? :)
>
> On Thu, Sep 6, 2012 at 2:47 PM, Jameson Li <[EMAIL PROTECTED]> wrote:
> > I want to test version hadoop-0.22.
> > But when configurate the fairescheduler, I have some troublesome. The
> > fairscheduler is not efficient.
> > And I have configured this items in the mapred-site.xml, and also I hava
> > copy the fairscheduler jar file to the $HADOOP_HOME/lib:
> >
> >   <property>
> >     <name>mapreduce.jobtracker.taskScheduler</name>
> >     <value>org.apache.hadoop.mapred.FairScheduler</value>
> >   </property>
> >
> > <property>
> >         <name>mapred.fairscheduler.allocation.file</name>
> >         <value>conf/pools.xml</value>
> > </property>
> > <property>
> >         <name>mapred.fairscheduler.preemption</name>
> >         <value>true</value>
> > </property>
> > <property>
> >         <name>mapred.fairscheduler.assignmultiple</name>
> >         <value>true</value>
> > </property>
> > <property>
> >         <name>mapred.fairscheduler.poolnameproperty</name>
> >         <value>mapred.queue.name</value>
> >         <description>job.set("mapred.queue.name",pool); // pool is set
> to
> > either 'high' or 'low' </description>
> > </property>
> > <property>
> >         <name>mapred.queue.names</name>
> >         <value>default,aaa,bbb</value>
> > </property>
> >
> > And the pools.xml in $HADOOP_HOME/conf 's content:
> >
> > <?xml version="1.0"?>
> > <allocations>
> >   <pool name="putindb">
> >    <minMaps>72</minMaps>
> >    <minReduces>16</minReduces>
> >    <maxRunningJobs>20</maxRunningJobs>
> >    <weight>3.0</weight>
> >    <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
> >   </pool>
> >
> >   <pool name="machinelearning">
> >    <minMaps>9</minMaps>
> >    <minReduces>2</minReduces>
> >    <maxRunningJobs>10</maxRunningJobs>
> >    <weight>2.0</weight>
> >    <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
> >   </pool>
> >
> >   <pool name="default">
> >    <minMaps>9</minMaps>
> >    <minReduces>2</minReduces>
> >    <maxRunningJobs>10</maxRunningJobs>
> >    <weight>1.0</weight>
> >    <minSharePreemptionTimeout>60</minSharePreemptionTimeout>
> >   </pool>
> >
> >   <defaultMinSharePreemptionTimeout>60</defaultMinSharePreemptionTimeout>
> >   <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout>
> > </allocations>
> >
> > Can someone help me?
> >
> >
> > 专注于Mysql,MSSQL,Oracle,Hadoop
>
>
>
> --
> Harsh J
>