Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project

Copy link to this message
Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project
That would be wonderful to have. +1  I would love to see MR run on more
then just HDFS/YARN.  So people can pick what execution environment makes
since for them, just like what MPI does, or something like what HDFS does
with FileSystem.  My perspective was just from the current state of
things, if we want to invert the relationship that fixes the problem.  I
would be happy to help with doing that.


On 8/31/12 12:06 PM, "Todd Lipcon" <[EMAIL PROTECTED]> wrote:

>On Fri, Aug 31, 2012 at 9:58 AM, Robert Evans <[EMAIL PROTECTED]> wrote:
>> The problem there is that YARN depends on Common, and MapReduce depends
>> YARN, so we would either have a circular dependency or we would have to
>> split off MapRedcue too.
>I haven't been in the MR codebase much of late, so I'll defer to your
>judgment here: would it be feasible to have an abstraction layer for
>"cluster manager" separated out into a pile of interfaces? Then we
>could leave MR inside Hadoop, and Yarn would have an "MR->Yarn
>binding" module. I'm not sure where the line would be drawn, but one
>possibility would be to separate out the MR _task_ code from the MR
>scheduling code (AM, Job Submission, etc)
>Again would be a large project, but as Eli said, it would help make MR
>more "relocatable" onto other cluster schedulers like Mesos (or even a
>traditional grid scheduler). Another possible boon there would be
>something I've discussed with Arun a few times: it would be cool if we
>could get the new MR task code (in particular the rewritten reduce,
>but also some of the new exciting work that Tsuyoshi and Mariappan are
>doing) running in the context of an MR1 cluster.
>> On 8/31/12 11:54 AM, "Eli Collins" <[EMAIL PROTECTED]> wrote:
>>>How about a proposal to just spin YARN off as a TLP?  Rationale:
>>>1. YARN started as a separate project and has a more independent
>>>community than Common/HDFS/MR (per below these communities do not
>>>divide at sub-project boundaries) that appears to want to be even more
>>>2. YARN is technically much easier to separate from the rest of the
>>>code base (than separating Common and HDFS for example). Separating it
>>>out will also help accelerate other efforts like MR2 support for
>>>Apache Mesos.
>>>3. It side steps a number of thorny issues (how to handle branch-1,
>>>how to handle what Hadoop is wrt enforcing trademark, who to remove
>>>people from the Hadoop PMC, etc) that haven't been addressed in any of
>>>these proposals.
>>>4. It's a proof point - if you can't make the case for YARN then
>>>there's no way we're going to make a case for splitting the other
>>>projects (this thread).
>>>Ie this doesn't have to be an all-or-nothing proposition for all
>>>sub-projects, since the communities don't fall on sub-project
>>>On Tue, Aug 28, 2012 at 7:33 PM, Mattmann, Chris A (388J)
>>><[EMAIL PROTECTED]> wrote:
>>>> [decided to minimize traffic and to simply put this in one thread]
>>>> Hi Guys,
>>>> See the recent discussion on these threads:
>>>> YARN as its own Hadoop "sub project": http://s.apache.org/WW1
>>>> Maintain a single committer list for the Hadoop project:
>>>> ...and just pay attention to the Hadoop project over the last 3-4
>>>>years. It's operating
>>>> as a single project, that's masking separate communities that
>>>>themselves are really
>>>> separate ASF projects.
>>>> At the ASF, this has been a problem area called "umbrella" projects
>>>>over the years,
>>>> all I've seen from them is wasted bandwidth, artificial barriers and
>>>>the inventions of
>>>> new ways to perform process mongering and to reduce the fun in
>>>>developing software
>>>> at this fantastic foundation.
>>>> I've talked about umbrella projects enough. We've diverted
>>>> Enough people have tried to act like there is some technical mumbo