Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: Fair Scheduler is not Fair why?


+
Justin Workman 2013-01-16, 15:45
Copy link to this message
-
Re: Fair Scheduler is not Fair why?
Nan Zhu 2013-01-16, 16:21
have you enabled task preemption?

Best,

--
Nan Zhu
School of Computer Science,
McGill University

On Wednesday, 16 January, 2013 at 10:45 AM, Justin Workman wrote:

> Looks like weight for both pools is equal and all map slots are used. Therefore I don't believe anyone has priority for the next slots. Try setting research weight to 2. This should allow research to take slots as tech released them.
>
> Sent from my iPhone
>
> On Jan 16, 2013, at 8:26 AM, Dhanasekaran Anbalagan <[EMAIL PROTECTED] (mailto:[EMAIL PROTECTED])> wrote:
>
> > HI Guys
> >
> > We configured fair scheduler with cdh4, Fair scheduler not work properly.
> > Map Task Capacity = 1380
> > Reduce Task Capacity = 720
> >
> > We create two users tech and research, we configured equal weight 1 But, I stared job in research user mapper will not allocated why?
> > please guide me guys.
> >
> > <?xml version="1.0"?>
> > <allocations>
> > <pool name="tech">
> >   <minMaps>5</minMaps>
> >   <minReduces>5</minReduces>
> >   <maxRunningJobs>30</maxRunningJobs>
> >   <weight>1.0</weight>
> > </pool>
> > <pool name="research">
> >   <minMaps>5</minMaps>
> >   <minReduces>5</minReduces>
> >   <maxRunningJobs>30</maxRunningJobs>
> >   <weight>1.0</weight>
> > </pool>
> > </allocations>
> >
> > Note: we have tested with Hadoop Stream job.
> >
> > Fair Scheduler Administration
> > Pools
> > Pool
> > Running Jobs
> > Map Tasks
> > Reduce Tasks
> > Scheduling Mode
> >
> > Min Share
> > Max Share
> > Running
> > Fair Share
> > Min Share
> > Max Share
> > Running
> > Fair Share
> >
> > research
> > 1
> > 5
> > -
> > 90
> > 690.0
> > 5
> > -
> > 0
> > 0.0
> > FAIR
> >
> > tech
> > 3
> > 5
> > -
> > 1266
> > 690.0
> > 5
> > -
> > 24
> > 24.0
> > FAIR
> >
> > default
> > 0
> > 0
> > -
> > 0
> > 0.0
> > 0
> > -
> > 0
> > 0.0
> > FAIR
> >
> >
> >
> > Running Jobs
> > Submitted
> > JobID
> > User
> > Name
> > Pool
> > Priority
> > Map Tasks
> > Reduce Tasks
> >
> > Finished
> > Running
> > Fair Share
> > Weight
> > Finished
> > Running
> > Fair Share
> > Weight
> >
> > Jan 16, 08:51
> > job_201301071639_2118 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2118)
> > tech
> > streamjob5335328828469969152.jar
> >
> >
> > 30466 / 53724
> > 583
> > 313.5
> > 1.0
> > 0 / 24
> > 0
> > 0.0
> > 1.0
> >
> > Jan 16, 09:56
> > job_201301071639_2147 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2147)
> > research
> > streamjob8832181817213433660.jar
> >
> >
> > 4175 / 9581
> > 90
> > 690.0
> > 1.0
> > 0 / 24
> > 0
> > 0.0
> > 1.0
> >
> > Jan 16, 10:01
> > job_201301071639_2148 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2148)
> > tech
> > streamjob8773848575543653055.jar
> >
> >
> > 1842 / 15484
> > 620
> > 313.5
> > 1.0
> > 0 / 24
> > 0
> > 0.0
> > 1.0
> >
> > Jan 16, 10:08
> > job_201301071639_2155 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2155)
> > tech
> > counterfactualsim-prod.eagle-EagleDepthSignalDisabled-prod.eagle
> >
> >
> > 387 / 450
> > 63
> > 63.0
> > 1.0
> > 0 / 24
> > 24
> > 24.0
> > 1.0
> >
> >
> >
> >
> >
> >
> > --
> >  
> >  
> >  

+
Dhanasekaran Anbalagan 2013-01-16, 18:07
+
Jeff Bean 2013-01-17, 18:35