Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Submitting a Hadoop task from withing a reducer


Copy link to this message
-
Re: Submitting a Hadoop task from withing a reducer
Thanks guys. I'll dig a little in Yarn.

On Tue, Oct 4, 2011 at 6:28 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:

> Yes. The reason I pointed it to him was that it seems like he's trying
> to do something with Hadoop for which MapReduce may not be right
> execution model. Yarn/MRv2 gives you the ability to try other
> execution models. As you pointed out, it may require some extra
> development, but it is more flexible than straight MapReduce and
> probably a better option in the long run over submitting jobs from
> inside of jobs.
>
> -Joey
>
> On Tue, Oct 4, 2011 at 12:11 PM, GOEKE, MATTHEW (AG/1000)
> <[EMAIL PROTECTED]> wrote:
> > Joey,
> >
> > Is yarn just a synonym for MRv2? And if so he would still have to create
> a custom application master for his job type right?
> >
> > Matt
> >
> > -----Original Message-----
> > From: Joey Echeverria [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, October 04, 2011 11:06 AM
> > To: [EMAIL PROTECTED]
> > Subject: Re: Submitting a Hadoop task from withing a reducer
> >
> > You may want to check out Yarn, coming in Hadoop 0.23:
> >
> > https://issues.apache.org/jira/browse/MAPREDUCE-279
> >
> > -Joey
> >
> > On Tue, Oct 4, 2011 at 11:45 AM, Yaron Gonen <[EMAIL PROTECTED]>
> wrote:
> >> Hi,
> >> Hadoop tasks are always stacked to form a linear user-managed workflow
> (a
> >> reduce step cannot start before all previous mappers have stopped etc).
> This
> >> may be problematic in recursive tasks: for example in a BFS we will not
> get
> >> any output until the longest branch has been reached.
> >> In order to solve than, an idea came up of submitting a whole Hadoop
> task
> >> from within a reducer. Have anyone tried it?
> >> Thanks.
> >>
> >
> >
> >
> > --
> > Joseph Echeverria
> > Cloudera, Inc.
> > 443.305.9434
> > This e-mail message may contain privileged and/or confidential
> information, and is intended to be received only by persons entitled
> > to receive such information. If you have received this e-mail in error,
> please notify the sender immediately. Please delete it and
> > all attachments from any servers, hard drives or any other media. Other
> use of this e-mail by you is strictly prohibited.
> >
> > All e-mails and attachments sent and received are subject to monitoring,
> reading and archival by Monsanto, including its
> > subsidiaries. The recipient of this e-mail is solely responsible for
> checking for the presence of "Viruses" or other "Malware".
> > Monsanto, along with its subsidiaries, accepts no liability for any
> damage caused by any such code transmitted by or accompanying
> > this e-mail or any attachment.
> >
> >
> > The information contained in this email may be subject to the export
> control laws and regulations of the United States, potentially
> > including but not limited to the Export Administration Regulations (EAR)
> and sanctions regulations issued by the U.S. Department of
> > Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of
> this information you are obligated to comply with all
> > applicable U.S. export laws and regulations.
> >
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>