Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - Heads up - merge branch-trunk-win to trunk


Copy link to this message
-
Re: Heads up - merge branch-trunk-win to trunk
Suresh Srinivas 2013-02-08, 17:12
I was planning to add details about the testing in the subsequent voting
thread.

As pointed out in the emails above, a lot of work is related to handling
difference
between the platforms related to paths and utilities. These changes lend
themselves
to be tested very well by the existing unit tests.

That said, a lot of testing has happened to validate the development done
in these
branches.

On branch-1-win, Hortonworks QA has been running a suite of comprehensive
system tests. This involves testing the branch with all the other stack
components,
such as Pig, Hive, HCat, HBase and Oozie. These tests have been done both
on
Linux and Windows. The testing has also been done both on JDK 6 and 7. In
addition,
as Mahadevan pointed out it has also been tested by early  customers with
production loads and  at scale.

branch-trunk-win has lot of code in common with branch-1-win and hence
benefits
from all the above testing. One place where additional work and testing was
done
in branch-trunk-win is related to YARN. All the MapReduce related work
loads have
been validated on single node cluster, cluster sizes with nodes upwards of
10.
On Thu, Feb 7, 2013 at 6:46 PM, Eli Collins <[EMAIL PROTECTED]> wrote:

> Thanks for the update Suresh.  Has any testing been done on the branch on
> Linux aside from running the unit tests?
>
> Thanks,
> Eli
>
>
> On Thu, Feb 7, 2013 at 5:42 PM, Suresh Srinivas <[EMAIL PROTECTED]
> >wrote:
>
> > The support for Hadoop on Windows was proposed in
> > HADOOP-8079<https://issues.apache.org/jira/browse/HADOOP-8079> almost
> > a year ago. The goal was to make Hadoop natively integrated,
> full-featured,
> > and performance and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in this
> > regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in
> > CHANGES.txt<
> >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.branch-1-win.txt?view=markup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk.
> > Merge patch for this is available on
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These changes
> > handle differences in platforms related to path names, process/task
> > management etc.
> > 2. Addition of winutils tools for managing file permissions and
> ownership,
> > user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> and
> > process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh,
> > start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and getting
> > ready for a merge. Currently the merge patch is passing close to 100% of
> > unit tests on Linux. Soon I will call for a vote to merge this branch
> into
> > trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows and
> how
> > to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
>

--
http://hortonworks.com/download/