-Re: Automatically mapping a job submitted by a particular user to a specific hadoop map-reduce queue
This capability currently does not exist in the fair scheduler (or other
schedulers, as far as I know), but a JIRA has been filed recently that
addresses a similar need. Would
https://issues.apache.org/jira/browse/MAPREDUCE-5132 work for what you're
trying to do? If not, would you mind filing a new JIRA for the
functionality you'd want?
On Wed, Apr 24, 2013 at 6:22 PM, Sagar Mehta <[EMAIL PROTECTED]> wrote:
> Hi Guys,
> We have a general purpose Hive cluster [about 200 nodes] which is used for
> various jobs like
> - Production
> - Experimental/Research
> - Adhoc queries
> We are using the fair-share scheduler to schedule them and for this we
> have corresponding 3 pools in the scheduler.
> *Here is what we want.*
> *A hive query submitted by a user with user-name A should go to one of
> the pools above based on a pre-defined mapping. We are wondering where/how
> to specify this mapping?*
> *We can do this manually by adding -Dmapred.job.queue.name="X" on a
> particular job run.*
> This puts the job on the map-reduce queue named "X" and the following
> configuration in the fair-share scheduler
> maps this to a pool named "X" in the fair-share scheduler.
> However we [while wearing our Hadoop developer/admin hat] don't want the
> user/analyst to specify that so as to enforce some cluster-use policy.
> Based on his/her username we want to automatically select which hadoop
> queue and subsequently which fair-share scheduler pool, his/her job should
> go to. I'm pretty sure this is a common use-case and wondering how to do
> this in Hadoop.
> Any help/insights/pointers would be greatly appreciated.
> PS - Btw we are using Cloudera cdh3u2 and the user jobs are Hive queries.