Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Hadoop JARs and Eclipse


Copy link to this message
-
RE: Hadoop JARs and Eclipse
Harsh,

Thanks for your response.  I don't need the whole of Hadoop projects in Eclipse, I just need to be able to build Hadoop and step into its source files.  Please help me better understand what you recommend as I have a very cursory understanding of maven and its interaction with Eclipse.  
-- From where will it fetch the Hadoop JARs?
-- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
-- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?

My goals are to build an application on Hadoop 2.0.  Currently I am using the 2.0.4 alpha release.  Eventually I this application will include an ApplicationMaster, auxiliary service, and a pile of embedded code and data.   However, I don't believe that I will be able to use Hadoop releases as-is without trouble.  I suspect that the following will be necessary at some point:
-- Step into the source code of referenced Hadoop code in the debugger.  
-- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
-- Switch to building from SVN instead of a release snapshot, and apply patches.

Thanks again,
John

-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <[EMAIL PROTECTED]>
Subject: Re: Hadoop JARs and Eclipse

If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?

The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:

<dependencies>
    <dependency>
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
      <version>2.0.4-alpha</version>
    </dependency>
  </dependencies>

Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".

On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <[EMAIL PROTECTED]> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse.  Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1)      Tell Eclipse about all of the Hadoop JARs
>
> 2)      Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse?  They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB