MapReduce, mail # user - server side DAG of jobs in YARN

server side DAG of jobs in YARN

Radim Kolar 2012-07-30, 19:29

Is there some support for server side control of DAG jobs? I mean
something which in Tool creates Jobs and submits them with list of
dependencies and then exits without waiting for end of jobs. I currently
have this implemented in client side, i have to wait until all jobs are
finished before System.exit().

JobControl and ControlledJob is client side thing. I think about Spring
Batch, it has some limited support for server side DAG.

Re: server side DAG of jobs in YARN

Bo Wang 2012-07-30, 21:33

Hi Radim,

Thanks for the mail. I am working on the server side DAG execution
actually. Here is a JIRA.

https://issues.apache.org/jira/browse/MAPREDUCE-4495

Thanks,
Bo

On Mon, Jul 30, 2012 at 12:29 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:

>
> Is there some support for server side control of DAG jobs? I mean
> something which in Tool creates Jobs and submits them with list of
> dependencies and then exits without waiting for end of jobs. I currently
> have this implemented in client side, i have to wait until all jobs are
> finished before System.exit().
>
> JobControl and ControlledJob is client side thing. I think about Spring
> Batch, it has some limited support for server side DAG.
>
>
>
>

Re: server side DAG of jobs in YARN

Bo Wang 2012-07-30, 22:18

Hi Radim,

Thanks for the mail. I am working on the server side DAG execution
actually. Here is a JIRA.

https://issues.apache.org/jira/browse/MAPREDUCE-4495

Thanks,
Bo

On Mon, Jul 30, 2012 at 12:29 PM, Radim Kolar <[EMAIL PROTECTED]> wrote:

>
> Is there some support for server side control of DAG jobs? I mean something which in Tool creates Jobs and submits them with list of dependencies and then exits without waiting for end of jobs. I currently have this implemented in client side, i have to wait until all jobs are finished before System.exit().
>
> JobControl and ControlledJob is client side thing. I think about Spring Batch, it has some limited support for server side DAG.
>
>

Re: server side DAG of jobs in YARN

Radim Kolar 2012-07-31, 10:15

> I am working on the server side DAG execution actually. Here is a JIRA.
>
> https://issues.apache.org/jira/browse/MAPREDUCE-4495
do you have some code in hand?

Re: server side DAG of jobs in YARN

Bo Wang 2012-08-01, 18:13

Not working yet. I expect to finish it in one month or so.

On Tue, Jul 31, 2012 at 3:15 AM, Radim Kolar <[EMAIL PROTECTED]> wrote:

>
> I am working on the server side DAG execution actually. Here is a JIRA.
>>
>> https://issues.apache.org/**jira/browse/MAPREDUCE-4495<https://issues.apache.org/jira/browse/MAPREDUCE-4495>
>>
> do you have some code in hand?
>