Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: Fair Scheduler is not Fair why?


Copy link to this message
-
Re: Fair Scheduler is not Fair why?
Dhanasekaran Anbalagan 2013-01-16, 16:32
HI Nan,

We have not enabled Fair Scheduler Preemption.

-Dhanasekaran.

Did I learn something today? If not, I wasted it.
On Wed, Jan 16, 2013 at 11:21 AM, Nan Zhu <[EMAIL PROTECTED]> wrote:

>  have you enabled task preemption?
>
> Best,
>
> --
> Nan Zhu
> School of Computer Science,
> McGill University
>
>
> On Wednesday, 16 January, 2013 at 10:45 AM, Justin Workman wrote:
>
> Looks like weight for both pools is equal and all map slots are used.
> Therefore I don't believe anyone has priority for the next slots. Try
> setting research weight to 2. This should allow research to take slots as
> tech released them.
>
> Sent from my iPhone
>
> On Jan 16, 2013, at 8:26 AM, Dhanasekaran Anbalagan <[EMAIL PROTECTED]>
> wrote:
>
>  HI Guys
>
> We configured fair scheduler with cdh4, Fair scheduler not work properly.
> Map Task Capacity = 1380
> Reduce Task Capacity = 720
>
> We create two users tech and research, we configured equal weight 1 But, I
> stared job in research user mapper will not allocated why?
> please guide me guys.
>
> <?xml version="1.0"?>
> <allocations>
> <pool name="tech">
>   <minMaps>5</minMaps>
>   <minReduces>5</minReduces>
>   <maxRunningJobs>30</maxRunningJobs>
>   <weight>1.0</weight>
> </pool>
> <pool name="research">
>   <minMaps>5</minMaps>
>   <minReduces>5</minReduces>
>   <maxRunningJobs>30</maxRunningJobs>
>   <weight>1.0</weight>
> </pool>
> </allocations>
>
> Note: we have tested with Hadoop Stream job.
>
> Fair Scheduler Administration Pools PoolRunning JobsMap TasksReduce TasksScheduling
> Mode Min ShareMax ShareRunningFair ShareMin ShareMax ShareRunningFair
> Share research15-90690.05-00.0FAIR tech35-1266690.05-2424.0FAIR default00-
> 00.00-00.0FAIR Running Jobs SubmittedJobIDUserNamePoolPriorityMap TasksReduce
> Tasks FinishedRunningFair ShareWeightFinishedRunningFair ShareWeight Jan
> 16, 08:51 job_201301071639_2118<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2118>
> tech streamjob5335328828469969152.jar   30466 / 53724583313.5 1.0 0 / 240
> 0.0 1.0 Jan 16, 09:56 job_201301071639_2147<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2147>
> research streamjob8832181817213433660.jar   4175 / 958190690.0 1.0 0 / 240
> 0.0 1.0 Jan 16, 10:01 job_201301071639_2148<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2148>
> tech streamjob8773848575543653055.jar   1842 / 15484620313.5 1.0 0 / 240
> 0.0 1.0 Jan 16, 10:08 job_201301071639_2155<http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2155>
> tech counterfactualsim-prod.eagle-EagleDepthSignalDisabled-prod.eagle   387
> / 4506363.0 1.0 0 / 242424.0 1.0
>
> --
>
>
>
>
>
>
+
Nan Zhu 2013-01-16, 16:43
+
Nan Zhu 2013-01-16, 16:50
+
Jeff Bean 2013-01-16, 17:02