Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Is there a way to limit # of hadoop tasks per user at runtime?


Copy link to this message
-
Re: Is there a way to limit # of hadoop tasks per user at runtime?
Sandy Ryza 2013-05-21, 21:57
Hi Mehmet,

Are you using MR1 or MR2?

The fair scheduler, present in both versions, but configured slightly
differently, allows you to limit the number of map and reduce tasks in a
queue.  The configuration can be updated at runtime by modifying the
scheduler's allocations file.  It also has a feature that automatically
maps jobs to queues based on the user submitted them.

Here are links to documentation in MR1 and MR2:
http://hadoop.apache.org/docs/stable/fair_scheduler.html
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

-Sandy

On Tue, May 21, 2013 at 2:43 PM, Mehmet Belgin <[EMAIL PROTECTED]
> wrote:

> Hi Everyone,
>
> I was wondering if there is a way for limiting the number of tasks
> (map+reduce) *per user* at runtime? Using an environment variable perhaps?
> I am asking this from a resource provisioning perspective. I am trying to
> come up with a N-token licensing system for multiple users to use our
> limited hadoop resources simultaneously. That is, when user A checks out 6
> tokens,  he/she can only run 6 hadoop tasks.
>
> If there is no such thing in hadoop, has anyone tried to integrate hadoop
> with torque/moab (or any other RM or scheduler)? Any advice in that
> direction will be appreciated :)
>
> Thanks in advance,
> -Mehmet
>
>
>
>
>
>
>