Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - I need help building Hadoop 2.0 for windows


Copy link to this message
-
Re: I need help building Hadoop 2.0 for windows
Steve Lewis 2014-01-17, 02:41
1) I have included the text when I build as described in the document

2) I included my patches to a couple of files in the hadoop jars which fix
issues - essentially Hadoop tries to set permissions which I find is really
unneeded under Windows and the fix is not to do that if running under
windows. I add these to my project earlier in the classpath than the
standard Hadoop jars and things run well without needing winutils.exe or
hadoop.dll

3) The real issue is that my experience is that jobs need to be debugged in
single node mode on a local machine before testing on the cluster. There
are issues which only emerge on the cluster but most of the logic can be
tested with the support of good debugging tools on a single machine.

4) A larger issue may be that on the cluster bash calls are handled by
forking. If Hadoop jobs require significant memory, mine need 2-3 gb, the
fork requires the same memory and I get failures when multiple jobs fork at
the same time. These failures are not repeatable but if jobs are large
enough there is a significant probability the job fails when a failure
happens 3 times. A good long term fix is to use a small server on the
slaves to handle bash callouts.
On Thu, Jan 16, 2014 at 10:00 AM, Chris Nauroth <[EMAIL PROTECTED]>wrote:

> Steve, I'm responding to your email from 1/6/2014 asking for help with
> building on Windows.
>
>
> http://mail-archives.apache.org/mod_mbox/hadoop-mapreduce-user/201401.mbox/%3CCALEj8ePTj6hZjqA%3D8sVG6J-%2BvMOYNn15vcYv1aAA%3DYhP-gKLwQ%40mail.gmail.com%3E
>
> I lost the original thread, so I'm pasting the full contents from the
> archived message below.
>
> Sorry that this isn't available as a binary distribution yet.  That's
> definitely a TODO.
>
> It sounds like you've already followed the instructions in BUILDING.txt
> and had a successful build for winutils.exe, but not hadoop.dll.  Can you
> provide more details about the problem you're seeing with the hadoop.dll
> build?
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
> I usually run Hadoop on a linux cluster but do most of my development in
> single machine mode under windows.
> This was fairly straightforward for 0.2. For 1.0 I needed to copy and fix
> FileUtils but for 2.0 I am expected to build 2 files from source -
> WinUtils.exe and hadoop.dll. There is really only ONE serious windows
> configuration: 64 bit intel and NO good reason why these files could not be
> available in a binary distribution -
> Is anyone using Hadoop 2.X and developing under windows whi can help.
> My configuration builds WunUtils.exe but not hadoop.dll and fails trying to
> set permission on a staging
> file file:/tmp/hadoop-Steve/mapred/staging/Steve2116067144/.staging
>
> I could use help and advice
>
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com