Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> RE: mvn eclipse:eclipse failure on windows


Copy link to this message
-
Re: mvn eclipse:eclipse failure on windows
At this point, running on Windows does require hadoop.dll in the library
path and winutils.exe accessible through HADOOP_HOME.  This differs from
Linux, where libhadoop.so is a soft dependency, and you can choose to run
without it.  (There are just too many things broken in subtle ways if we
don't interact properly with the native Windows APIs.)

I wonder if there is anything we can configure in our pom.xml files so that
the library path and HADOOP_HOME requirements get injected into the Eclipse
project files automatically when you run mvn eclipse:eclipse.  This would
make it easier for Windows users to import the project correctly.  I don't
use Eclipse anymore, so I'm not deeply familiar with the options, and I
didn't find anything helpful from my quick research.

BTW, the HADOOP_HOME requirement will change at some point.  HADOOP-9422
tracks removal of this dependency, so you might want to consider setting a
watch on that issue.

https://issues.apache.org/jira/browse/HADOOP-9422

Chris Nauroth
Hortonworks
http://hortonworks.com/

On Wed, Jul 17, 2013 at 9:43 PM, Uma Maheswara Rao G
<[EMAIL PROTECTED]>wrote:

> Hi Chris,
>
> mvn test works fine for me via commandline.
> But I am trying with eclipse.
>
> Looks like, 2 things missing in eclipse, 1. /hadoop-common/target/bin is
> not coming in library path  (just checked NativeLoader is not finding
> hadoop.dll)and 2. HADOOP_HOME_DIR is not set
> On setting this 2 things, things started working for me. I am not sure if
> there is something I missed here to do things automatic. Before win merge I
> was not having such dependencies.
>
> Regards,
> Uma
>
> -----Original Message-----
> From: Chris Nauroth [mailto:[EMAIL PROTECTED]]
> Sent: 17 July 2013 05:13
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: mvn eclipse:eclipse failure on windows
>
> Loading hadoop.dll in tests is supposed to work via a common shared
> maven-surefire-plugin configuration that sets the PATH environment variable
> to include the build location of the dll:
>
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-project/pom.xml#L894
>
> (On Windows, the shared library path is controlled with PATH instead of
> LD_LIBRARY_PATH on Linux.)
>
> This configuration has been working fine in all of the dev environments
> I've seen, but I'm wondering if something different is happening in your
> environment.  Does your hadoop.dll show up in
> hadoop-common-project/hadoop-common/target/bin?  Is there anything else
> that looks unique in your environment?
>
> Also, another potential gotcha is the Windows max path length limitation of
> 260 characters.  Deeply nested project structures like Hadoop can cause
> very long paths for the built artifacts, and you might not be able to load
> the files if the full path exceeds 260 characters.  The workaround for now
> is to keep the codebase in a very short root folder.  (I use C:\hdc .)
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jul 15, 2013 at 1:07 PM, Chuan Liu <[EMAIL PROTECTED]> wrote:
>
> > Hi Uma,
> >
> > I suggest you do a 'mvn install -DskipTests' before running 'mvn
> > eclipse:eclipse'.
> >
> > Thanks,
> > Chuan
> >
> > -----Original Message-----
> > From: Uma Maheswara Rao G [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, July 12, 2013 7:42 PM
> > To: [EMAIL PROTECTED]
> > Cc: [EMAIL PROTECTED]
> > Subject: Re: mvn eclipse:eclipse failure on windows
> >
> > HI Chris,
> >   eclipse:eclipse works but still I am seeing UnsatisfiesLinkError.
> > Explicitly I pointed java.library.path to where hadoop.dll geneated. This
> > dll generated with my clean install command only.   My pc is 64 but and
> > also set Platform=x64 while building. But does not help.
> >
> > Regards,
> > Uma
> >
> >
> >
> >
> >
> >
> > On Fri, Jul 12, 2013 at 11:45 PM, Chris Nauroth
> > <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Uma,
> > >
> > > I just tried getting a fresh copy of trunk and running "mvn clean