Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Using CapacityScheduler to divide resources between jobs (not users)


Copy link to this message
-
Using CapacityScheduler to divide resources between jobs (not users)
Amit Sela 2013-07-06, 14:07
Hi all,

I'm running Hadoop 1.0.4 on a modest cluster (~20 machines).
The jobs running on the cluster can be divided (resource wise) as follows:

1. Very short jobs: less then 1 minute.
2. Normal jobs: 2-3 minutes up to an hour or two.
3. Very long jobs: days of processing. (still not active and the reason for
my inquiries here).

I was thinking of using the CapacityScheduler and divide the cluster
resources something like:
1. min: