Hi all,

I have a beginner's question:
How can I configure yarn to allow multiple parallel mapreduce jobs to run?
Currently, the execution is sequential: each submitted job waits for the
previous to run.

In MR1, this could be done by enabling the fairscheduler/capacityscheduler.
Is there a similar configuration in Yarn as well?

Thanks a lot!
--
Bharath Ravi

Yes! , you can do the same in yarn as well.
-P

On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> I have a beginner's question:
> How can I configure yarn to allow multiple parallel mapreduce jobs to run?
> Currently, the execution is sequential: each submitted job waits for the
> previous to run.
>
> In MR1, this could be done by enabling the fairscheduler/capacityscheduler.
> Is there a similar configuration in Yarn as well?
>
> Thanks a lot!
> --
> Bharath Ravi
>

Thanks, Prashant!
I'll try Yarn out with the Fairscheduler.

On 8 November 2011 01:01, Prashant Sharma <[EMAIL PROTECTED]> wrote:

> Yes! , you can do the same in yarn as well.
> -P
>
> On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <[EMAIL PROTECTED]>
> wrote:
>
> > Hi all,
> >
> > I have a beginner's question:
> > How can I configure yarn to allow multiple parallel mapreduce jobs to
> run?
> > Currently, the execution is sequential: each submitted job waits for the
> > previous to run.
> >
> > In MR1, this could be done by enabling the
> fairscheduler/capacityscheduler.
> > Is there a similar configuration in Yarn as well?
> >
> > Thanks a lot!
> > --
> > Bharath Ravi
> >
>

--
Bharath Ravi

FairScheduler isn't ported yet to YARN. The default scheduler is there (
which is FifoScheduler) and CapacityScheduler can be configured too.

HTH,
+Vinod
On Wed, Nov 9, 2011 at 10:14 AM, Bharath Ravi <[EMAIL PROTECTED]>wrote:

> Thanks, Prashant!
> I'll try Yarn out with the Fairscheduler.
>
> On 8 November 2011 01:01, Prashant Sharma <[EMAIL PROTECTED]>
> wrote:
>
> > Yes! , you can do the same in yarn as well.
> > -P
> >
> > On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi all,
> > >
> > > I have a beginner's question:
> > > How can I configure yarn to allow multiple parallel mapreduce jobs to
> > run?
> > > Currently, the execution is sequential: each submitted job waits for
> the
> > > previous to run.
> > >
> > > In MR1, this could be done by enabling the
> > fairscheduler/capacityscheduler.
> > > Is there a similar configuration in Yarn as well?
> > >
> > > Thanks a lot!
> > > --
> > > Bharath Ravi
> > >
> >
>
>
>
> --
> Bharath Ravi
>

The configuration options are somewhat different for yarn, then they are for MRV1.  You probably want to generate the documentation for yarn

mvn site

And then read through it about how to set up your cluster

./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/target/site/index.html

There is documentation about setting up the capacity scheduler too.  If you run into any issues then reply here and if the documentation needs to be cleaned up was can file a JIRA against the documentation.  The documentation is kind of new so it would be good to get some real feedback on it.

--Bobby Evans

On 11/9/11 3:07 AM, "Vinod Kumar Vavilapalli" <[EMAIL PROTECTED]> wrote:

FairScheduler isn't ported yet to YARN. The default scheduler is there (
which is FifoScheduler) and CapacityScheduler can be configured too.

HTH,
+Vinod
On Wed, Nov 9, 2011 at 10:14 AM, Bharath Ravi <[EMAIL PROTECTED]>wrote:

> Thanks, Prashant!
> I'll try Yarn out with the Fairscheduler.
>
> On 8 November 2011 01:01, Prashant Sharma <[EMAIL PROTECTED]>
> wrote:
>
> > Yes! , you can do the same in yarn as well.
> > -P
> >
> > On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi all,
> > >
> > > I have a beginner's question:
> > > How can I configure yarn to allow multiple parallel mapreduce jobs to
> > run?
> > > Currently, the execution is sequential: each submitted job waits for
> the
> > > previous to run.
> > >
> > > In MR1, this could be done by enabling the
> > fairscheduler/capacityscheduler.
> > > Is there a similar configuration in Yarn as well?
> > >
> > > Thanks a lot!
> > > --
> > > Bharath Ravi
> > >
> >
>
>
>
> --
> Bharath Ravi
>

Thanks a lot all!
I'll try configuring the capacity scheduler on yarn, and get back to you if
I run into any issues.

On 9 November 2011 10:41, Robert Evans <[EMAIL PROTECTED]> wrote:

> The configuration options are somewhat different for yarn, then they are
> for MRV1.  You probably want to generate the documentation for yarn
>
> mvn site
>
> And then read through it about how to set up your cluster
>
>
> ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/target/site/index.html
>
> There is documentation about setting up the capacity scheduler too.  If
> you run into any issues then reply here and if the documentation needs to
> be cleaned up was can file a JIRA against the documentation.  The
> documentation is kind of new so it would be good to get some real feedback
> on it.
>
> --Bobby Evans
>
> On 11/9/11 3:07 AM, "Vinod Kumar Vavilapalli" <[EMAIL PROTECTED]>
> wrote:
>
> FairScheduler isn't ported yet to YARN. The default scheduler is there (
> which is FifoScheduler) and CapacityScheduler can be configured too.
>
> HTH,
> +Vinod
>
>
> On Wed, Nov 9, 2011 at 10:14 AM, Bharath Ravi <[EMAIL PROTECTED]
> >wrote:
>
> > Thanks, Prashant!
> > I'll try Yarn out with the Fairscheduler.
> >
> > On 8 November 2011 01:01, Prashant Sharma <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Yes! , you can do the same in yarn as well.
> > > -P
> > >
> > > On Tue, Nov 8, 2011 at 3:24 AM, Bharath Ravi <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have a beginner's question:
> > > > How can I configure yarn to allow multiple parallel mapreduce jobs to
> > > run?
> > > > Currently, the execution is sequential: each submitted job waits for
> > the
> > > > previous to run.
> > > >
> > > > In MR1, this could be done by enabling the
> > > fairscheduler/capacityscheduler.
> > > > Is there a similar configuration in Yarn as well?
> > > >
> > > > Thanks a lot!
> > > > --
> > > > Bharath Ravi
> > > >
> > >
> >
> >
> >
> > --
> > Bharath Ravi
> >
>
>
--
Bharath Ravi