Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # dev >> RE: mvn eclipse:eclipse failure on windows


Copy link to this message
-
Re: mvn eclipse:eclipse failure on windows
At this point, running on Windows does require hadoop.dll in the library
path and winutils.exe accessible through HADOOP_HOME.  This differs from
Linux, where libhadoop.so is a soft dependency, and you can choose to run
without it.  (There are just too many things broken in subtle ways if we
don't interact properly with the native Windows APIs.)

I wonder if there is anything we can configure in our pom.xml files so that
the library path and HADOOP_HOME requirements get injected into the Eclipse
project files automatically when you run mvn eclipse:eclipse.  This would
make it easier for Windows users to import the project correctly.  I don't
use Eclipse anymore, so I'm not deeply familiar with the options, and I
didn't find anything helpful from my quick research.

BTW, the HADOOP_HOME requirement will change at some point.  HADOOP-9422
tracks removal of this dependency, so you might want to consider setting a
watch on that issue.

https://issues.apache.org/jira/browse/HADOOP-9422

Chris Nauroth
Hortonworks
http://hortonworks.com/

On Wed, Jul 17, 2013 at 9:43 PM, Uma Maheswara Rao G
<[EMAIL PROTECTED]>wrote:

> Hi Chris,
>
> mvn test works fine for me via commandline.
> But I am trying with eclipse.
>
> Looks like, 2 things missing in eclipse, 1. /hadoop-common/target/bin is
> not coming in library path  (just checked NativeLoader is not finding
> hadoop.dll)and 2. HADOOP_HOME_DIR is not set
> On setting this 2 things, things started working for me. I am not sure if
> there is something I missed here to do things automatic. Before win merge I
> was not having such dependencies.
>
> Regards,
> Uma
>
> -----Original Message-----
> From: Chris Nauroth [mailto:[EMAIL PROTECTED]]
> Sent: 17 July 2013 05:13
> To: [EMAIL PROTECTED]
> Cc: [EMAIL PROTECTED]
> Subject: Re: mvn eclipse:eclipse failure on windows
>
> Loading hadoop.dll in tests is supposed to work via a common shared
> maven-surefire-plugin configuration that sets the PATH environment variable
> to include the build location of the dll:
>
>
> https://github.com/apache/hadoop-common/blob/trunk/hadoop-project/pom.xml#L894
>
> (On Windows, the shared library path is controlled with PATH instead of
> LD_LIBRARY_PATH on Linux.)
>
> This configuration has been working fine in all of the dev environments
> I've seen, but I'm wondering if something different is happening in your
> environment.  Does your hadoop.dll show up in
> hadoop-common-project/hadoop-common/target/bin?  Is there anything else
> that looks unique in your environment?
>
> Also, another potential gotcha is the Windows max path length limitation of
> 260 characters.  Deeply nested project structures like Hadoop can cause
> very long paths for the built artifacts, and you might not be able to load
> the files if the full path exceeds 260 characters.  The workaround for now
> is to keep the codebase in a very short root folder.  (I use C:\hdc .)
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jul 15, 2013 at 1:07 PM, Chuan Liu <[EMAIL PROTECTED]> wrote:
>
> > Hi Uma,
> >
> > I suggest you do a 'mvn install -DskipTests' before running 'mvn
> > eclipse:eclipse'.
> >
> > Thanks,
> > Chuan
> >
> > -----Original Message-----
> > From: Uma Maheswara Rao G [mailto:[EMAIL PROTECTED]]
> > Sent: Friday, July 12, 2013 7:42 PM
> > To: [EMAIL PROTECTED]
> > Cc: [EMAIL PROTECTED]
> > Subject: Re: mvn eclipse:eclipse failure on windows
> >
> > HI Chris,
> >   eclipse:eclipse works but still I am seeing UnsatisfiesLinkError.
> > Explicitly I pointed java.library.path to where hadoop.dll geneated. This
> > dll generated with my clean install command only.   My pc is 64 but and
> > also set Platform=x64 while building. But does not help.
> >
> > Regards,
> > Uma
> >
> >
> >
> >
> >
> >
> > On Fri, Jul 12, 2013 at 11:45 PM, Chris Nauroth
> > <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Uma,
> > >
> > > I just tried getting a fresh copy of trunk and running "mvn clean
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB