Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> [Vote] Merge branch-trunk-win to trunk


Copy link to this message
-
Re: [Vote] Merge branch-trunk-win to trunk
Similar personal concern as Robert: Does this bring about a
development process change? Do new features all need to work on
Windows as well to go into trunk (i.e. immediately or eventually,
either way requires a new policy for all of us devs)? Not that anyone
would be avoiding doing that, I just ask cause it could impact time
and effort required for any major undertaking.

While most of the project today is cross-platform (via Java, etc.,
which thankfully remove path handling problems and the sorts),
performance improvements at least are going the native side these
days, which is where I see this have some impact. We've not been
perfectly successful in having the natives continuously work on
Solaris/etc. in the past, mainly due to the platform focus of the
majority (not all) of devs working on the project(s). Some form of a
development policy here would ensure proper Windows support for the
features we intend to ship along are not very divergent such that we
end up having to maintain docs as well, detailing each task yet to be
done (these tend to grow if allowed).

Useful to also note from another OSS project KDE, that their working
builds of Windows are usually on 1-2 releases of the past (i.e. for
example, KDE release is currently at 4.10, but the last released
Windows port is still 4.8 today). KDE uses Qt, which is cross-platform
by itself, but there's still a port team and a ported release
maintained separately (but under the same org.) due to the major
development happening on Linux. Same case for *BSD as well. My own
patches there at some point have caused trouble cause I did something
that I only tested on one platform, and about a bit later things got
revised to support the other ones where it was unnecessarily breaking.

Or if am being too concerned about feature/performance divergence, let me know.

On Wed, Feb 27, 2013 at 9:47 PM, Robert Evans <[EMAIL PROTECTED]> wrote:
> After this is merged in is Windows still going to be a second class
> citizen but happens to work for more than just development or is it a
> fully supported platform where if something breaks it can block a release?
>  How do we as a community intend to keep Windows support from breaking?
> We don't have any Jenkins slaves to be able to run nightly tests to
> validate everything still compiles/runs.  This is not a blocker for me
> because we often rely on individuals and groups to test Hadoop, but I do
> think we need to have this discussion before we put it in.
>
> --Bobby
>
> On 2/26/13 4:55 PM, "Suresh Srinivas" <[EMAIL PROTECTED]> wrote:
>
>>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>>I
>>am happy to announce that we are ready for the merge.
>>
>>Here is a brief recap on the highlights of the work done:
>>- Command-line scripts for the Hadoop surface area
>>- Mapping the HDFS permissions model to Windows
>>- Abstracted and reconciled mismatches around differences in Path
>>semantics
>>in Java and Windows
>>- Native Task Controller for Windows
>>- Implementation of a Block Placement Policy to support cloud
>>environments,
>>more specifically Azure.
>>- Implementation of Hadoop native libraries for Windows (compression
>>codecs, native I/O)
>>- Several reliability issues, including race-conditions, intermittent test
>>failures, resource leaks.
>>- Several new unit test cases written for the above changes
>>
>>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>>ported from branch-1-win to a branch based on trunk.
>>
>>For details of the testing done, please see the thread -
>>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>>https://issues.apache.org/jira/browse/HADOOP-8562>.
>>
>>This was a large undertaking that involved developing code, testing the
>>entire Hadoop stack, including scale tests. This is made possible only

Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB