Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # dev - windows support in trunk?


Copy link to this message
-
RE: windows support in trunk?
Bikas Saha 2013-09-25, 22:00
Please don’t have Cygwin on the path while running Hadoop. Its not needed
and might actually interfere in case there is some Unix functionality that
is still missing from the Windows utility that is compiled for Hadoop. Only
sh and tar are required for building and unit testing. Win32 versions of
these are available freely.

All sh scripts for Hadoop should have cmd equivalents. Please file a jira if
you find one that’s missing. Usually porting them is simple and patches are
welcome!
You should be able to follow the same flow to install and bring up the
cluster as you do with Unix.

You will find that a winutils.exe file is built in your hadoop build and
this is required to be able to run Hadoop on Windows. It translates Unix
like file permissions and provides other Win32 native functionality for the
Java code. IIRC this exe needs to be available in the %HADOOP_HOME%/bin
directory.

Bikas
-----Original Message-----
From: Andrey Klochkov [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, September 25, 2013 11:30 AM
To: common-dev
Subject: Re: windows support in trunk?

I don't have problems with building. I have issues with running it. Usually
I do this with the following steps:
1. build using "dist" maven profile to generate the distro under
hadoop-dist/target 2. set HADOOP_HOME pointing to
hadoop-dist/target/hadoop-x.x.x-SNAPSHOT
3. fix configs under $HADOOP_HOME/etc
4. use start-dfs.sh, start-yarh.sh and mr-jobhistory-daemon.sh to start the
daemons

The "dist" profile doesn't work on Windows without Cygwin as it depends on
shell and tar utilities. How can I overcome this?

Actually I installed Cygwin and managed to do step #1. Then I succeed with
steps 2-4 and started all daemons using *cmd scripts. The only issue I have
is with the jobhistory daemon, as there's no mr-jobhistory-daemon.cmd
script. Is there an alternative way to start it?

On Wed, Sep 25, 2013 at 11:12 AM, Arpit Agarwal
<[EMAIL PROTECTED]>wrote:

> Andrey, thanks for testing your patch on Windows.
>
> cygwin is no longer required. Instead, please install the free Windows
> SDK and build from an SDK command prompt. This is documented in
> BUILDING.txt.
> Please feel free to email the dev list if you get stuck at any step.
>
>
> On Tue, Sep 24, 2013 at 10:31 PM, Bikas Saha <[EMAIL PROTECTED]>
> wrote:
>
> > BUILDING.txt in the source code has a section for building on Windows.
> >
> > Bikas
> >
> > -----Original Message-----
> > From: Andrey Klochkov [mailto:[EMAIL PROTECTED]]
> > Sent: Tuesday, September 24, 2013 5:11 PM
> > To: common-dev
> > Subject: Re: windows support in trunk?
> >
> > Great, thanks!
> >
> > One more question on Windows support: how do I create a distribution
> > and run it? It's for dev purposes - testing a patch to work on
> > Windows, so I
> can't
> > really use HDP. On Linux/OSX I just do a build with -Pdist and then
> > just use startXXX scripts from hadoop-project-dist/target to run all
> > the daemons.
> > Are
> > there any docs on how to do it on Windows? E.g. do I need cygwin? I
> > can build sources without cygwin, but hadoop-project-dist requires
> > shell executable to build the tarball.
> >
> >
> > On Tue, Sep 24, 2013 at 3:31 PM, Arun C Murthy <[EMAIL PROTECTED]>
> > wrote:
> >
> > >
> > > On Sep 24, 2013, at 3:24 PM, Andrey Klochkov
> > > <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Is Windows support in trunk currently? Or should I still use
> > > > trunk-win to experiment with Hadoop on Windows? I've seen number
> > > > of windows related patches going into trunk that's why I'm asking.
> Thanks!
> > > >
> > > > I know I just need to ask Chris Nauroth, but sending here just
> > > > in case
> > > it's
> > > > useful for others.
> > >
> > > *smile*
> > >
> > > Yes, FTR, Windows support is in trunk, branch-2 and branch-2.1.
> > >
> > > Arun
> > >
> > >
> > > >
> > > > --
> > > > Andrey Klochkov
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >

Andrey Klochkov
Grid Dynamics
Skype: aklochkov_gd
www.griddynamics.com
[EMAIL PROTECTED]

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.