Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop fair scheduler doubt: allocate jobs to pool


Copy link to this message
-
Re: Hadoop fair scheduler doubt: allocate jobs to pool
>From the fairscheduler docs I assume the following should work:

<property>
 <name>mapred.fairscheduler.poolnameproperty</name>
   <value>pool.name</value>
</property>

<property>
  <name>pool.name</name>
      <value>${mapreduce.job.group.name}</value>
</property>

which means that the default pool will be the group of the user that has
submitted the job. In your case I think that allocations.xml is correct. If
you want to explicitly define a job to specific pool from your
allocation.xml file you can define it as follows:

Configuration conf3 = conf;
conf3.set("pool.name", "pool3"); // conf.set(propriety.name, value)

Let me know if it works..
On 29 February 2012 14:18, Austin Chungath <[EMAIL PROTECTED]> wrote:

> How can I set the fair scheduler such that all jobs submitted from a
> particular user group go to a pool with the group name?
>
> I have setup fair scheduler and I have two users: A and B (belonging to the
> user group hadoop)
>
> When these users submit hadoop jobs, the jobs from A got to a pool named A
> and the jobs from B go to a pool named B.
>  I want them to go to a pool with their group name, So I tried adding the
> following to mapred-site.xml:
>
> <property>
>  <name>mapred.fairscheduler.poolnameproperty</name>
> <value>group.name</value>
> </property>
>
> But instead the jobs now go to the default pool.
> I want the jobs submitted by A and B to go to the pool named "hadoop". How
> do I do that?
> also how can I explicity set a job to any specified pool?
>
> I have set the allocation file (fair-scheduler.xml) like this:
>
> <allocations>
>  <pool name="hadoop">
>    <minMaps>1</minMaps>
>    <minReduces>1</minReduces>
>    <maxMaps>3</maxMaps>
>    <maxReduces>3</maxReduces>
>  </pool>
>  <userMaxJobsDefault>5</userMaxJobsDefault>
> </allocations>
>
> Any help is greatly appreciated.
> Thanks,
> Austin
>