+1 non-binding.

I have extensively tested this on both Windows and Linux over the last few
months.

Thanks,
-Arpit

On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <[EMAIL PROTECTED]> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <[EMAIL PROTECTED]> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><[EMAIL PROTECTED]>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a

>
> I'm concerned about the above. Personally, I don't have access to any
> Windows boxes with development tools, and I know nothing about developing
> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
> for powerpoint :)
> If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
> am I supposed to proceed?
>
I think a reasonable compromise would be that the tests should always
> *build* on Windows before commit, and contributors should do their best to
> look at the test logs for any Windows-specific failures. But, beyond
> looking at the logs, a "-1 Tests failed on windows" should not block a
> commit.
>

I think these concerns may be exaggerated.

I have attached a patch to the jira:
https://issues.apache.org/jira/secure/attachment/12571338/branch-trunk-win-min.patch
The number of lines goes to 1537 lines from the original patch with 15958
lines.

So the choice of Java in Hadoop meant that the work required was minimal.
Most
of the changes are related platform specific behavior such File separator
and line
separator which are fairly trivial to debug and fix.

+1 non-binding

I am playing with it for several months in a multi-node Windows cluster environment and found it is very stable. I am sure that it can help us to bring more developers like me (JAVA & Windows Developers) to contribute more and help the Hadoop customer & developer communities.

Thanks,
Kanna

-----Original Message-----
From: Raja Aluri [mailto:[EMAIL PROTECTED]]
Sent: Thursday, February 28, 2013 11:17 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: [Vote] Merge branch-trunk-win to trunk

+1 non-binding
Nice to see that this work is going to trunk.

Raja  Aluri
On Tue, Feb 26, 2013 at 2:55 PM, Suresh Srinivas <[EMAIL PROTECTED]>wrote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb
> 8th. I am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent
> test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt -
> Common changes<http://bit.ly/Xe7Ynv>, HDFS
> changes<http://bit.ly/13QOSo9>, and YARN and MapReduce changes
> <http://bit.ly/128zzMt>. This is the work ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread -
> http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562< https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing
> the entire Hadoop stack, including scale tests. This is made possible
> only with the contribution from many many folks in the community.
> Following people contributed to this work: Ivan Mitic, Chuan Liu,
> Ramya Sunil, Bikas Saha, Kanna Karanam, John Gordon, Brandon Li, Chris
> Nauroth, David Lao, Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz,
> Mike Liddell, Jing Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja
> Aluri, Giridharan Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp,
> Arun Murthy, Tsz-Wo Nicholas Sze, Suresh Srinivas and Sanjay Radia.
> There are many others who contributed as well providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <[EMAIL PROTECTED]>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> merged
> > to trunk. Having Windows as one of the supported Hadoop platforms is
> > a fantastic opportunity both for the Hadoop project and Microsoft
> customers.
> >
> > This work began around a year back when a few of us started with a
> > basic port of Hadoop on Windows. Ever since, the Hadoop team in
> > Microsoft have made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression
> > codecs, native I/O) - Several reliability issues, including

+1 (binding)

Very exciting to see this effort come to fruition!

Arun

On Feb 26, 2013, at 2:55 PM, Suresh Srinivas wrote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
> am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path semantics
> in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud environments,
> more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent test
> failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt -
> Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
> and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread -
> http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing the
> entire Hadoop stack, including scale tests. This is made possible only with
> the contribution from many many folks in the community. Following people
> contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
> Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
> Srinivas and Sanjay Radia. There are many others who contributed as well
> providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <[EMAIL PROTECTED]>wrote:
>
>> It is super exciting to look at the prospect of these changes being merged
>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>> fantastic opportunity both for the Hadoop project and Microsoft customers.
>>
>> This work began around a year back when a few of us started with a basic
>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>> made significant progress in the following areas:
>> (PS: Some of these items are already included in Suresh's email, but
>> including again for completeness)
>>
>> - Command-line scripts for the Hadoop surface area
>> - Mapping the HDFS permissions model to Windows
>> - Abstracted and reconciled mismatches around differences in Path
>> semantics in Java and Windows
>> - Native Task Controller for Windows
>> - Implementation of a Block Placement Policy to support cloud
>> environments, more specifically Azure.
>> - Implementation of Hadoop native libraries for Windows (compression
>> codecs, native I/O) - Several reliability issues, including
>> race-conditions, intermittent test failures, resource leaks.
>> - Several new unit test cases written for the above changes
>>
>> In the process, we have closely engaged with the Apache open source
>> community and have got great support and assistance from the community in
>> terms of contributing fixes, code review comments and commits.
>>
>> In addition, the Hadoop team at Microsoft has also made good progress in
>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of
>> these changes have already been committed to the respective trunks with
>> help from various committers and contributors. It is great to see the

Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

+1
Java has done the bulk of the work in making Hadoop multi-platform.
Windows specific code is a tiny percentage of the code.
Jeninks support for windows is going help us keep the platform portable going forward.
I expect that the vast majority of new commits have  no problems. I propose that we start by fixing problems that Jenkins raises but not block new commits for too long if the author does not have a windows box or if a volunteer does not step up.

sanjay

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy as our development is with Java. If we have some native code portings, then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting for some performance improvements and he is interested only in Linux. Then I hope some other contributors will help in getting the windows patch if possible.
If others busy to get that done with in time lines, then I think we can commit Linux support patch and leave one JIRA open for Windows support? [ make sure that porting options configurable and platform check and give release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [[EMAIL PROTECTED]]
Sent: Wednesday, February 27, 2013 4:25 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th. I
am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path semantics
in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud environments,
more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only with
the contribution from many many folks in the community. Following people
contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
Srinivas and Sanjay Radia. There are many others who contributed as well
providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh
On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<[EMAIL PROTECTED]>wrote:

> It is super exciting to look at the prospect of these changes being merged
> to trunk. Having Windows as one of the supported Hadoop platforms is a
> fantastic opportunity both for the Hadoop project and Microsoft customers.
>
> This work began around a year back when a few of us started with a basic
> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
> made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
http://hortonworks.com/download/

> I have attached a patch to the jira:
> https://issues.apache.org/jira/secure/attachment/12571338/branch-trunk-win-min.patch
> The number of lines goes to 1537 lines from the original patch with 15958
> lines.

Suresh, this might be a confusing statement as your patch includes
only Yarn changes.
Could you please update it.

Thanks,
--Konst

On Wed, Feb 27, 2013 at 6:17 PM, Suresh Srinivas <[EMAIL PROTECTED]> wrote:
>>
>> I'm concerned about the above. Personally, I don't have access to any
>> Windows boxes with development tools, and I know nothing about developing
>> on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
>> for powerpoint :)
>
>
>> If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
>> am I supposed to proceed?
>>
> I think a reasonable compromise would be that the tests should always
>> *build* on Windows before commit, and contributors should do their best to
>> look at the test logs for any Windows-specific failures. But, beyond
>> looking at the logs, a "-1 Tests failed on windows" should not block a
>> commit.
>>
>
> I think these concerns may be exaggerated.
>
> I have attached a patch to the jira:
> https://issues.apache.org/jira/secure/attachment/12571338/branch-trunk-win-min.patch
> The number of lines goes to 1537 lines from the original patch with 15958
> lines.
>
> So the choice of Java in Hadoop meant that the work required was minimal.
> Most
> of the changes are related platform specific behavior such File separator
> and line
> separator which are fairly trivial to debug and fix.

On Mar 1, 2013, at 1:57 PM, Konstantin Shvachko wrote:

> Commitment is a good thing.
> I think the two builds that I proposed are a prerequisite for Win support.
> If we commit windows patch people will start breaking it the next day.
> Which we wont know without the nightly build and wont be able to fix
> without the on-demand one.

They clearly are a prerequisite for declaring "official" support for windows.
But they should not be a prerequisite for the merge,.
Currently we enable windows through cygwin. There is no jenkins. Folks have been  fixing  windows issues as they are discovered.
Merging the branch makes the situation no worse than it is today - all tests pass on Linux, there is no regression.
Merging now removes  the cygwin dependency.

Jenkins is critical to make windows officially supported platform without cygwin.
When Jenkins is enabled, the team that has worked on this branch will have to fix any bugs that have arisen in the mean time.

sanjay

Hi Konstantin S,

I accidentally merged some windows bug fixes to branch-2.  However, branch-2 did not compile -- it showed that the windows changes were missing in branch-2.  The ones causing compilation problems were already reverted (Thanks Sid and Suresh.)  Sorry for the inconvenience. 
Tsz-Wo
________________________________
 From: Konstantin Shvachko <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Tuesday, March 26, 2013 5:25 AM
Subject: Re: [Vote] Merge branch-trunk-win to trunk
 
Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin
On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <[EMAIL PROTECTED]> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> [EMAIL PROTECTED]>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <[EMAIL PROTECTED]>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > [EMAIL PROTECTED]>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <[EMAIL PROTECTED]
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would, or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit