Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Jobtracker fair scheduler


Copy link to this message
-
Re: Jobtracker fair scheduler
Thanks Anurag. I will check and keep you posted in case of any issue.
On Nov 22, 2013 12:54 AM, "anurag tangri" <[EMAIL PROTECTED]> wrote:

> Here's step-by-step on how to do this:
> ===============>
> Here's step by step on how to do this:
>
> 1. Add following in mapred-site.xml:
>
>
>   <property>
>     <name>mapred.fairscheduler.preemption</name>
>     <final>true</final>
>     <value>true</value>
>   </property>
>   <property>
>     <name>mapred.jobtracker.taskScheduler</name>
>     <final>true</final>
>     <value>org.apache.hadoop.mapred.FairScheduler</value>
>   </property>
>   <property>
>     <name>mapred.acls.enabled</name>
>     <final>true</final>
>     <value>true</value>
>   </property>
>     <property>
>     <name>mapred.fairscheduler.allow.undeclared.pools</name>
>     <final>true</final>
>     <value>false</value>
>   </property>
>   <property>
>     <name>mapred.fairscheduler.poolnameproperty</name>
>     <final>true</final>
>     <value>mapred.job.queue.name</value>
>   </property>
>   <property>
>     <name>mapred.fairscheduler.allocation.file</name>
>     <final>true</final>
>     <value>/etc/hadoop/conf/allocations.xml</value>
>   </property>
>   <property>
>     <name>mapred.queue.names</name>
>     <final>true</final>
>     <value>sqoop,default</value>
>   </property>
>   <property>
>     <name>mapreduce.job.acl-view-job</name>
>     <final>true</final>
>     <value>*</value>
>   </property>
>
>
> 2. Create mapred-queue-acls.xml in same dir as mapred-site.xml where you
> define all the queues:
>
> <configuration>
>   <property>
>     <name>mapred.queue.sqoop.acl-submit-job</name>
>     <value>usera, userb</value>
>   </property>
>   <property>
>     <name>mapred.queue.sqoop.acl-administer-jobs</name>
>     <value>usera, userc</value>
>   </property>
> <! above two properties for every queue defined in mapred-site.xml -->
> </configuration>
>
>
> 3. Define fair scheduler allocations file (location specified in
> maped-site.xml) to use the queues defined and assign resources to them:
>
> <?xml version="1.0"?>
> <allocations>
>   <defaultMinSharePreemptionTimeout>300</defaultMinSharePreemptionTimeout>
>   <pool name="sqoop">
>     <minMaps>700</minMaps>
>     <minReduces>175</minReduces>
>     <maxRunningJobs>25</maxRunningJobs>
>   </pool>
>   <pool name="default">
>     <minMaps>120</minMaps>
>     <minReduces>40</minReduces>
>     <maxRunningJobs>40</maxRunningJobs>
>   </pool>
>   <fairSharePreemptionTimeout>600</fairSharePreemptionTimeout>
> </allocations>
>
>
> 4. After this, restart job tracker.
>
>
> And your queues should be in place. You can then submit job to right queue
> using
> -Dmapred.job.queue.name=sqoop or default
>
> And check the job going to actual queue at the link:
> http://<job tracker hostname>:50030/scheduler
>
>
> Look at the property mapreduce.job.acl-view-job in mapred-site. Means
> anyone can view the jobs.
>
>
> Try above and let us know any issues you still face.
>
>
> ===============================>
>
> Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
>   On Thursday, November 21, 2013 11:21 AM, anurag tangri <
> [EMAIL PROTECTED]> wrote:
>  Hi Viswanathan,
> What steps have you followed to set fair scheduler ?
>
> Thanks,
>  Anurag Tangri
>
>
> Never wear your best trousers when you go out to fight for freedom and
> truth.- Henrik Ibsen
>
>
>   On Thursday, November 21, 2013 10:25 AM, Viswanathan J <
> [EMAIL PROTECTED]> wrote:
>  Hi,
> I'm running hadoop with 1.2.1, all my jobs are running in single queue
> (Queue 1) only all the time. But I have configured default, queue 1&2.
> Why jobs are not scheduled to all the queues.
> Please help. Running like this will be any issue?
> Thanks,
>
>
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB