[Moving to mapreduce-dev, copying common-dev]

Hi,

On Thu, Sep 9, 2010 at 11:30 AM, radheshyam nanduri
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am working on writing a scheduler plugin for Hadoop.

Currently, the model supported to plug-in schedulers to Hadoop is to
extend the TaskScheduler class in o.a.h.mapred package. Primarily what
a 'plug-in' scheduler can do is that given a set of jobs and a
tasktracker, it can assign one or more suitable tasks to the
tasktracker. The scheduler will have flexibility in choosing the job
and the tasks it wants to schedule. You can take a look at some of the
existing schedulers like CapacityTaskScheduler or FairScheduler to see
what they do and how.

> I have divided the job received into number of tasks.

This is already done in the framework when a job is submitted. Are you
overriding this ? Can you explain what you are doing in some more
detail ?

> My task now is to assign a task on to a particular TaskTracker.
> I want to start the Task right away with a method which accepts the Task and
> TaskTracker as arguments.

I am not sure I am following this. A task needs to be sent via
Hadoop's RPC mechanisms to a tasktracker where it should be executed.
So, conceptually, it is the tasktracker that has an RPC method which
accepts tasks to launch. The task is launched typically straight-away,
but in case of certain scheduling choices, it could have to wait for a
short while to get a free slot to execute.

> Could you guide me on doing this.

It may be better if you can describe what you want the plug-in
scheduler to achieve.

Thanks
Hemanth

>
>
> Thanks in advance.
>
>
> --
> Radheshyam Nanduri
>

hi,

Thanks for the reply.

The problem i want to solve is, I am writing a scheduler plugin in which, i
want to divide a job into tasks and assign to TaskTrackers(Obvious).

Then, i want to assign the tasks on to which ever TaskTracker i want and
assign it whenever i want.

I am using
org.apache.hadoop.mapred.TaskScheduler.assignTasks(TaskTrackerStatus
taskTrackerStatus) which would return list of Tasks, on every Heart Beat.

In the above method, i am calling
org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask method to get Task
for a job. But everytime it is called, i would only get a null as return.

Initially i thought that JobStatus.status is not set to RUNNING.
But the problem is not that.

Is there any other way that i could assign tasks on to a TaskTracker apart
from above method.
hoping a solution from you.

Thank you

Radheshyam Nanduri

On Fri, Sep 10, 2010 at 6:46 PM, Hemanth Yamijala <[EMAIL PROTECTED]>wrote:

> [Moving to mapreduce-dev, copying common-dev]
>
> Hi,
>
> On Thu, Sep 9, 2010 at 11:30 AM, radheshyam nanduri
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I am working on writing a scheduler plugin for Hadoop.
>
> Currently, the model supported to plug-in schedulers to Hadoop is to
> extend the TaskScheduler class in o.a.h.mapred package. Primarily what
> a 'plug-in' scheduler can do is that given a set of jobs and a
> tasktracker, it can assign one or more suitable tasks to the
> tasktracker. The scheduler will have flexibility in choosing the job
> and the tasks it wants to schedule. You can take a look at some of the
> existing schedulers like CapacityTaskScheduler or FairScheduler to see
> what they do and how.
>
> > I have divided the job received into number of tasks.
>
> This is already done in the framework when a job is submitted. Are you
> overriding this ? Can you explain what you are doing in some more
> detail ?
>
> > My task now is to assign a task on to a particular TaskTracker.
> > I want to start the Task right away with a method which accepts the Task
> and
> > TaskTracker as arguments.
>
> I am not sure I am following this. A task needs to be sent via
> Hadoop's RPC mechanisms to a tasktracker where it should be executed.
> So, conceptually, it is the tasktracker that has an RPC method which
> accepts tasks to launch. The task is launched typically straight-away,
> but in case of certain scheduling choices, it could have to wait for a
> short while to get a free slot to execute.
>
> > Could you guide me on doing this.
>
> It may be better if you can describe what you want the plug-in
> scheduler to achieve.
>
> Thanks
> Hemanth
>
> >
> >
> > Thanks in advance.
> >
> >
> > --
> > Radheshyam Nanduri
> >
>

--
Radheshyam Nanduri