Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> pig job priority control


Copy link to this message
-
Re: pig job priority control
On trunk:
set mapred.job.queue.name myFastLaneQueue ;

either on grunt or in script.

In 0.7 or earlier through -D switch at command line at the time of
invocation of pig:
-Dmapred.job.queue.name=myFastLaneQueue

Hope it helps,
Ashutosh

On Mon, Jun 28, 2010 at 13:29, jiang licht <[EMAIL PROTECTED]> wrote:
> Thanks, Jeff. How to submit a pig job to a queue used by these schedulers? I think a more general question is how to specify a customized job configuration for a pig job (in addition to specifying which queue to submit this job).
>
> Thanks,
>
> Michael
>
> --- On Mon, 6/21/10, Jeff Zhang <[EMAIL PROTECTED]> wrote:
>
> From: Jeff Zhang <[EMAIL PROTECTED]>
> Subject: Re: pig job priority control
> To: [EMAIL PROTECTED]
> Date: Monday, June 21, 2010, 8:14 PM
>
> You can also change the task scheduler of hadoop .
> Please refer http://hadoop.apache.org/common/docs/r0.20.0/fair_scheduler.html
> http://hadoop.apache.org/common/docs/r0.20.0/capacity_scheduler.html
>
>
> On Tue, Jun 22, 2010 at 12:57 AM, jiang licht <[EMAIL PROTECTED]> wrote:
>> What is the best way to manage multiple pig jobs such that they can get chance to run simultaneously? W/o priority control, some job will block other jobs (a small job with e.g. a mapper and a reducer will have to wait for its turn). This is bad. For example, some job with big number of mappers (the number is bigger than the maximum number mappers that are allowed to run in the cluster) will consume all resources and other jobs submitted later will have to wait to load their mappers until it is done.
>>
>> Thanks!
>>
>>
>>
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
>
>
>