Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> development environment for hadoop core

Copy link to this message
Re: development environment for hadoop core
On 01/15/2013 06:50 PM, Erik Paulson wrote:
> Hello -
> I'm curious what Hadoop developers use for their day-to-day hacking on
> Hadoop. I'm talking changes to the Hadoop libraries and daemons, and not
> developing Map-Reduce jobs or using using the HDFS Client libraries to talk
> to a filesystem from an application.
> I've checked out Hadoop, made minor changes and built it with Maven, and
> tracked down the resulting artifacts in a target/ directory that I could
> deploy. Is this typically how a cloudera/hortonworks/mapr/etc dev works, or
> are the IDEs more common?
I haven't built Hadoop yet myself.  Your use of "a" in "a target/
directory" indicates you're also kind of new with Maven itself, as
that's the standard output folder for any Maven project.  One of many
nice things about Maven is once you learn how to build one project with
it you pretty much know how to build any project with it, as
everything's standardized with it.

Probably best to stick with the command line for building and use
Eclipse for editing, to keep things simple, but don't forget the mvn
eclipse:eclipse command to set up Eclipse projects that you can
subsequently import into your Eclipse IDE:

> I realize this sort of sounds like a dumb question, but I'm mostly curious
> what I might be missing out on if I stay away from anything other than vim,
> and not being entirely sure where maven might be caching jars that it uses
> to build,

That will be your local Maven repository, in an .m2 hidden folder in
your user home directory.
> and how careful I have to be to ensure that my changes wind up in
> the right places without having to do a clean build every time.

Maven can detect changes (using mvn install instead of mvn clean
install), but I prefer doing clean builds.  You can use the
-Dmaven.test.skip setting to speed up your "mvn clean installs" if you
don't wish to run the tests each time.

> Thanks!
> -Erik
Glen Mazza
Talend Community Coders - coders.talend.com
blog: www.jroller.com/gmazza