Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop fair scheduler doubt: allocate jobs to pool


Copy link to this message
-
Re: Hadoop fair scheduler doubt: allocate jobs to pool
>From the fairscheduler docs I assume the following should work:

<property>
 <name>mapred.fairscheduler.poolnameproperty</name>
   <value>pool.name</value>
</property>

<property>
  <name>pool.name</name>
      <value>${mapreduce.job.group.name}</value>
</property>

which means that the default pool will be the group of the user that has
submitted the job. In your case I think that allocations.xml is correct. If
you want to explicitly define a job to specific pool from your
allocation.xml file you can define it as follows:

Configuration conf3 = conf;
conf3.set("pool.name", "pool3"); // conf.set(propriety.name, value)

Let me know if it works..
On 29 February 2012 14:18, Austin Chungath <[EMAIL PROTECTED]> wrote:

> How can I set the fair scheduler such that all jobs submitted from a
> particular user group go to a pool with the group name?
>
> I have setup fair scheduler and I have two users: A and B (belonging to the
> user group hadoop)
>
> When these users submit hadoop jobs, the jobs from A got to a pool named A
> and the jobs from B go to a pool named B.
>  I want them to go to a pool with their group name, So I tried adding the
> following to mapred-site.xml:
>
> <property>
>  <name>mapred.fairscheduler.poolnameproperty</name>
> <value>group.name</value>
> </property>
>
> But instead the jobs now go to the default pool.
> I want the jobs submitted by A and B to go to the pool named "hadoop". How
> do I do that?
> also how can I explicity set a job to any specified pool?
>
> I have set the allocation file (fair-scheduler.xml) like this:
>
> <allocations>
>  <pool name="hadoop">
>    <minMaps>1</minMaps>
>    <minReduces>1</minReduces>
>    <maxMaps>3</maxMaps>
>    <maxReduces>3</maxReduces>
>  </pool>
>  <userMaxJobsDefault>5</userMaxJobsDefault>
> </allocations>
>
> Any help is greatly appreciated.
> Thanks,
> Austin
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB