Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # dev >> Re: Launching Tasks From JobTracker


Copy link to this message
-
Re: Launching Tasks From JobTracker
[Moving to mapreduce-dev, copying common-dev]

Hi,

On Thu, Sep 9, 2010 at 11:30 AM, radheshyam nanduri
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am working on writing a scheduler plugin for Hadoop.

Currently, the model supported to plug-in schedulers to Hadoop is to
extend the TaskScheduler class in o.a.h.mapred package. Primarily what
a 'plug-in' scheduler can do is that given a set of jobs and a
tasktracker, it can assign one or more suitable tasks to the
tasktracker. The scheduler will have flexibility in choosing the job
and the tasks it wants to schedule. You can take a look at some of the
existing schedulers like CapacityTaskScheduler or FairScheduler to see
what they do and how.

> I have divided the job received into number of tasks.

This is already done in the framework when a job is submitted. Are you
overriding this ? Can you explain what you are doing in some more
detail ?

> My task now is to assign a task on to a particular TaskTracker.
> I want to start the Task right away with a method which accepts the Task and
> TaskTracker as arguments.

I am not sure I am following this. A task needs to be sent via
Hadoop's RPC mechanisms to a tasktracker where it should be executed.
So, conceptually, it is the tasktracker that has an RPC method which
accepts tasks to launch. The task is launched typically straight-away,
but in case of certain scheduling choices, it could have to wait for a
short while to get a free slot to execute.

> Could you guide me on doing this.

It may be better if you can describe what you want the plug-in
scheduler to achieve.

Thanks
Hemanth

>
>
> Thanks in advance.
>
>
> --
> Radheshyam Nanduri
>
+
radheshyam nanduri 2010-09-17, 08:30
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB