Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - RE: [Vote] Merge branch-trunk-win to trunk


Copy link to this message
-
Re: [Vote] Merge branch-trunk-win to trunk
Eric Baldeschwieler 2013-03-01, 04:47
+1 (non-binding)

A few of observations:

- Windows has actually been a supported platform for Hadoop since 0.1 .  Doug championed supporting windows then and we've continued to do it with varying vigor over time.  To my knowledge we've never made a decision to drop windows support.  The change here is improving our support and dropping the requirement of cigwin.  We had Nutch windows users on the list in 2006 and we've been supporting windows FS requirements since inception.

- A little pragmatism will go a long way.  As a community we've got to stay committed to keeping hadoop simple (so it does work on many platforms) and extending it to take advantage of key emerging OS/hardware features, such as containers, new FSs, virtualization, flash ...  We should all plan to let new features & optimizations emerge that don't work everywhere, if they are compelling and central to hadoop's mission of being THE best fabric for storing and processing big data.  

- A UI project like KDE has to deal with the MANY differences between windows and linux UI APIs.  Hadoop faces no such complex challenge and hence can be maintained from a single codeline IMO.  It is mostly abstracted from the OS APIs via Java and our design choices.  Where it is not we can continue to add plugable abstractions.

On Feb 28, 2013, at 6:01 PM, Matt Foley <[EMAIL PROTECTED]> wrote:

> +1 (binding)
>
> Apache is supposed to be about the community.  We have here a community of
> developers, who have actively and openly worked to add a major improvement
> to Hadoop: the ability to work cross-platform.  Furthermore, the size of
> the substantive part of the needed patch is only about 1500 lines, much
> smaller than quite a few other additions to Hadoop over the last few
> months.  We should welcome and support this change, and make sure that the
> code stays cross-platform going forward by extending our CI practices,
> especially pre-commit "test-patch", to also include Windows.
>
> As most of you know, my colleague Giri Kesavan (PMC member) helps maintain
> the Linux CI capability for Hadoop.  I've talked with him, and he and I are
> committing to getting test-patch implemented for Windows, so that along
> with the current automated "+1"s required to commit, we can add two more,
> for javac build in Windows and core unit tests in Windows.
>
> Members of the team implementing cross-platform compatibility, including
> Microsoft employees, have opened the discussion for providing hardware or
> VM resources to perform this additional CI testing.  I will assist them to
> work with the Apache Infra team and figure out how to make it happen.
>
> I understand there is some concern about the additional platform test.
> My going-in
> presumption, based on Java's intrinsic, pretty-good, cross-platform
> compatibility, is that patches to Hadoop will by default also have
> cross-platform compatibility, unless they are written in an explicitly
> platform-dependent way.  I also believe that in the vast majority of cases
> the cross-platform compatibility of Java will carry thru to Hadoop patches,
> without additional effort on the developer's part.
>
> Let's try it, and see what happens.  If we actually find a frequent
> difficulty, we'll change to engineer around it.  But I believe that, in the
> rare cases where a Windows-specific failure occurs, there will be a number
> of people (new, enthusiastic members of the community! :-) willing to help.
> If such help is not forthcoming, then we can discuss work-arounds, but
> like a previous poster, I am confident in the community.
>
> Regards,
> --Matt
>
>
>
> On Thu, Feb 28, 2013 at 12:21 PM, Chuan Liu <[EMAIL PROTECTED]> wrote:
>
>> +1 (non-binding)
>>>
>>> As someone also contributed to porting Hadoop to Windows, I think Java
>>> already provided a very good platform independent platform.
>>> For features that are not available in Java, we will try to provide our
>>> platform independent APIs that abstract OS tasks away.