Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # dev >> pluggable resources


+
Radim Kolar 2012-10-22, 10:41
+
Robert Evans 2012-10-22, 16:56
Copy link to this message
-
Re: pluggable resources
Agree with Bobby, baby steps

On Mon, Oct 22, 2012 at 9:56 AM, Robert Evans <[EMAIL PROTECTED]> wrote:
> I agree that having it be pluggable opens up a lot of new possibilities.
> +1 for the idea.  Although I think in the short term we are having enough
> problems as it is with just CPU and memory that it may be a little while
> before we get to a pluggable solution.  Once YARN-2 goes in, if you can
> get an initial proof of concept patch for a generic solution I would be
> happy to review it and push for it to go in.
>
> --Bobby
>
> On 10/22/12 5:41 AM, "Radim Kolar" <[EMAIL PROTECTED]> wrote:
>
>>I have proposal for improved resource scheduling.
>>
>>https://issues.apache.org/jira/browse/MAPREDUCE-4256
>>
>>as i see, development seems to go other way for example in
>>https://issues.apache.org/jira/browse/YARN-2 for every added kind of
>>resource there has to be significant rework.
>>
>>you do not see benefits of having framework able to handle custom
>>resource types? Its not all about memory and cores. You need to schedule
>>jobs based on other factors (network capacity, availability of GPU
>>cores, data locality).
>>
>>And every cluster might have special considerations for example do not
>>overload central SQL database. We usually have few hundred submitted
>>jobs, proper resource sharing is essential. No point in running jobs
>>which needs GPU which is in use by other mapper, better to run some
>>other jobs until gpu becomes available again.
>

--
Alejandro
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB