-Re: Is there a way to limit # of hadoop tasks per user at runtime?
Sandy Ryza 2013-05-21, 21:57
Are you using MR1 or MR2?
The fair scheduler, present in both versions, but configured slightly
differently, allows you to limit the number of map and reduce tasks in a
queue. The configuration can be updated at runtime by modifying the
scheduler's allocations file. It also has a feature that automatically
maps jobs to queues based on the user submitted them.
Here are links to documentation in MR1 and MR2:
On Tue, May 21, 2013 at 2:43 PM, Mehmet Belgin <[EMAIL PROTECTED]
> Hi Everyone,
> I was wondering if there is a way for limiting the number of tasks
> (map+reduce) *per user* at runtime? Using an environment variable perhaps?
> I am asking this from a resource provisioning perspective. I am trying to
> come up with a N-token licensing system for multiple users to use our
> limited hadoop resources simultaneously. That is, when user A checks out 6
> tokens, he/she can only run 6 hadoop tasks.
> If there is no such thing in hadoop, has anyone tried to integrate hadoop
> with torque/moab (or any other RM or scheduler)? Any advice in that
> direction will be appreciated :)
> Thanks in advance,