Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Hadoop JARs and Eclipse


Copy link to this message
-
RE: Hadoop JARs and Eclipse
Harsh,

Thanks for your response.  I don't need the whole of Hadoop projects in Eclipse, I just need to be able to build Hadoop and step into its source files.  Please help me better understand what you recommend as I have a very cursory understanding of maven and its interaction with Eclipse.  
-- From where will it fetch the Hadoop JARs?
-- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
-- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?

My goals are to build an application on Hadoop 2.0.  Currently I am using the 2.0.4 alpha release.  Eventually I this application will include an ApplicationMaster, auxiliary service, and a pile of embedded code and data.   However, I don't believe that I will be able to use Hadoop releases as-is without trouble.  I suspect that the following will be necessary at some point:
-- Step into the source code of referenced Hadoop code in the debugger.  
-- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
-- Switch to building from SVN instead of a release snapshot, and apply patches.

Thanks again,
John

-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <[EMAIL PROTECTED]>
Subject: Re: Hadoop JARs and Eclipse

If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?

The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:

<dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.0.4-alpha</version>
    </dependency>
  </dependencies>

Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".

On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <[EMAIL PROTECTED]> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse.  Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1)      Tell Eclipse about all of the Hadoop JARs
>
> 2)      Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse?  They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>

--
Harsh J