Radim Kolar 2012-10-22, 10:41
I have proposal for improved resource scheduling.
as i see, development seems to go other way for example in
https://issues.apache.org/jira/browse/YARN-2 for every added kind of
resource there has to be significant rework.
you do not see benefits of having framework able to handle custom
resource types? Its not all about memory and cores. You need to schedule
jobs based on other factors (network capacity, availability of GPU
cores, data locality).
And every cluster might have special considerations for example do not
overload central SQL database. We usually have few hundred submitted
jobs, proper resource sharing is essential. No point in running jobs
which needs GPU which is in use by other mapper, better to run some
other jobs until gpu becomes available again.