Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - [Vote] Merge branch-trunk-win to trunk

Copy link to this message
Re: [Vote] Merge branch-trunk-win to trunk
Robert Evans 2013-02-28, 19:29
My initial question was mostly intended to understand the desired new
classification of Windows after the merge, and how we plan to maintain
Windows support.  I am happy to hear that hardware for Jenkins will be
provided.  I am also fine, at least initially, with us trying to treat
Windows as a first class supported platform.  But I realize that there are
a lot of people that do not have easy access to Windows for
development/debugging, myself included. I also don't want to slow down the
pace of development too much because of this.  It will cause some
organizations that do not use or support Windows to be more likely to run
software that has diverged from an official release.  It also has the
potential to make the patch submission process even more difficult, which
increases the likelihood of submitters abandoning patches.  However, the
great thing about being in a community is we can change if we need to.

I am +0 for the merge.  I am not a Windows expert so I don't feel
comfortable giving it a true +1.

On 2/28/13 10:45 AM, "Chris Nauroth" <[EMAIL PROTECTED]> wrote:

>I'd like to share a few anecdotes about developing cross-platform,
>hopefully to address some of the concerns about adding overhead to the
>development process.  By reviewing past cases of cross-platform Linux vs.
>Windows bugs, we can get a sense for how the development process could
>in the future.
>HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
>Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
>committed on trunk covering the case of a local file system interaction on
>a file containing a ':'.  On Windows, ':' in a path has special meaning as
>part of the drive specifier (i.e. C:), so this test cannot pass when
>running on Windows.  In this kind of case, the cross-platform bug is
>obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
>this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
>HDFS-4274: BlockPoolSliceScanner does not close verification log during
>shutdown.  This caused problems for MiniDFSCluster-based tests running on
>Windows.  Failure to close the verification log meant that we didn't
>release file locks, so the tests couldn't delete/recreate working
>directories during teardown/setup.  Arguably, this was always a bug, and
>running on Windows just exposed it because of its stricter rules about
>locking.  This is a more complex fix, but it doesn't require
>platform-specific knowledge.  If some future patch accidentally regresses
>this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
>Jenkins.  Ideally, it would get fixed pre-commit, because it doesn't
>require Windows-specific knowledge.  There is also the matter of impact.
> Re-breaking this would re-break many test suites on Windows.
>HADOOP-9232: JniBasedUnixGroupsMappingWithFallback fails on Windows with
>UnsatisfiedLinkError.  This was introduced by HADOOP-8712, which switched
>to JniBasedUnixGroupsMappingWithFallback as the default
>hadoop.security.group.mapping, but did not provide a Windows
>of the JNI function.  In this case, there was a strong desire to get
>HADOOP-8712 into a release, fixing it on Windows required native Windows
>API knowledge, and Windows users had a simple workaround available by
>changing their configs back to ShellBasedUnixGroupsMapping.  I think this
>is the kind of situation where we could allow HADOOP-8712 to commit
>-1 from Windows Jenkins, with fairly quick follow-up from an engineer with
>the Windows expertise to fix it.
>To summarize, I don't think it needs to differ greatly from our current
>development process.  We're all responsible for breadth of understanding
>and maintenance of the whole codebase, but we also rely on specific
>individuals with deep expertise in particular areas for certain issues.
> Sometimes we commit despite a -1 from Jenkins, based on the community's