Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # dev - [Vote] Merge branch-trunk-win to trunk

Suresh Srinivas 2013-02-26, 22:55
Robert Evans 2013-02-27, 16:17
Eli Collins 2013-02-27, 19:56
Suresh Srinivas 2013-02-27, 22:54
Todd Lipcon 2013-02-27, 23:43
Ivan Mitic 2013-02-28, 02:32
John Gordon 2013-02-28, 11:25
Chris Nauroth 2013-02-28, 16:45
Robert Evans 2013-02-28, 19:29
Copy link to this message
Re: [Vote] Merge branch-trunk-win to trunk
Chris Nauroth 2013-02-28, 19:50
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).

I found 2 remaining TODOs introduced in the current merge patch.  One is in
ContainerLaunch.java.  The container launch script was trying to set a
CLASSPATH that exceeded the Windows maximum command line length.  The fix
was to wrap the long classpath into an intermediate jar containing only a
manifest file with a Class-Path entry.  (See YARN-316.)  Just to be
conservative, we wrapped this logic in an if (Shell.WINDOWS) guard and
marked a TODO to remove it later and use that approach on all platforms
after additional testing.  I've tested this code path successfully on Mac
too, but several people wanted additional testing and performance checks
before removing the if (Shell.WINDOWS) guard.  That work is tracked in an
existing jira: YARN-358.

The other TODO is for winutils to print more usage information and
examples.  At this point, I think winutils is printing sufficient
information, and we can just remove the TODO.  I just submitted a new jira
to start that conversation: HADOOP-9348.

Thank you,
On Thu, Feb 28, 2013 at 11:29 AM, Robert Evans <[EMAIL PROTECTED]> wrote:

> My initial question was mostly intended to understand the desired new
> classification of Windows after the merge, and how we plan to maintain
> Windows support.  I am happy to hear that hardware for Jenkins will be
> provided.  I am also fine, at least initially, with us trying to treat
> Windows as a first class supported platform.  But I realize that there are
> a lot of people that do not have easy access to Windows for
> development/debugging, myself included. I also don't want to slow down the
> pace of development too much because of this.  It will cause some
> organizations that do not use or support Windows to be more likely to run
> software that has diverged from an official release.  It also has the
> potential to make the patch submission process even more difficult, which
> increases the likelihood of submitters abandoning patches.  However, the
> great thing about being in a community is we can change if we need to.
> I am +0 for the merge.  I am not a Windows expert so I don't feel
> comfortable giving it a true +1.
> --Bobby
> On 2/28/13 10:45 AM, "Chris Nauroth" <[EMAIL PROTECTED]> wrote:
> >I'd like to share a few anecdotes about developing cross-platform,
> >hopefully to address some of the concerns about adding overhead to the
> >development process.  By reviewing past cases of cross-platform Linux vs.
> >Windows bugs, we can get a sense for how the development process could
> >look
> >in the future.
> >
> >HADOOP-9131: TestLocalFileSystem#testListStatusWithColons cannot run on
> >Windows.  As part of an earlier jira, HADOOP-8962, there was a new test
> >committed on trunk covering the case of a local file system interaction on
> >a file containing a ':'.  On Windows, ':' in a path has special meaning as
> >part of the drive specifier (i.e. C:), so this test cannot pass when
> >running on Windows.  In this kind of case, the cross-platform bug is
> >obvious, and the fix is obvious (assumeTrue(!Shell.WINDOWS)).  Ideally,
> >this would get fixed pre-commit after seeing a -1 from the Windows Jenkins
> >slave.
> >
> >HDFS-4274: BlockPoolSliceScanner does not close verification log during
> >shutdown.  This caused problems for MiniDFSCluster-based tests running on
> >Windows.  Failure to close the verification log meant that we didn't
> >release file locks, so the tests couldn't delete/recreate working
> >directories during teardown/setup.  Arguably, this was always a bug, and
> >running on Windows just exposed it because of its stricter rules about
> >file
> >locking.  This is a more complex fix, but it doesn't require
> >platform-specific knowledge.  If some future patch accidentally regresses
> >this, then we'll likely see +1 from Linux Jenkins and -1 from Windows
Harsh J 2013-02-27, 19:20
Konstantin Shvachko 2013-03-02, 03:14
Konstantin Shvachko 2013-03-01, 20:18
Chris Douglas 2013-03-01, 21:28
Konstantin Shvachko 2013-03-01, 21:57
Chris Douglas 2013-03-02, 00:57
Konstantin Boudnik 2013-03-01, 23:47
Suresh Srinivas 2013-03-02, 01:55
Konstantin Boudnik 2013-03-02, 03:33
Matt Foley 2013-03-02, 20:32
Konstantin Shvachko 2013-03-03, 03:32
Matt Foley 2013-03-03, 19:08
Konstantin Shvachko 2013-03-03, 20:16
Bikas Saha 2013-03-01, 17:55
Konstantin Boudnik 2013-03-01, 01:04
Ramya Sunil 2013-03-01, 00:59
Suresh Srinivas 2013-03-06, 19:50
Andrew Purtell 2013-03-25, 20:36
Suresh Srinivas 2013-03-26, 00:08