-Re: development environment for hadoop core
Steve Loughran 2013-01-16, 08:40
My setup ( I work from home)
# OS/X laptop w/ 30" monitor
# FTTC broadband, 55Mbit/s down, 15+ up -it's the upload bandwidth that
really helps development: http://www.flickr.com/photos/steve_l/8050751551/
# IntelliJ IDEA IDE, settings edited for a 2GB Heap
# Maven on the command line for builds
# I run a "mvn install -DskipTests" every morning to ensure that apache's
own -SNAPSHOT artifacts aren't pulled in.
# CentOS 6.3 VM for doing the full binary build & test, making my own RPMs,
One thing that annoys me is that I've got an airplay-driven hifi set up,
and during builds there's enough CPU/RAM load that the music has dropouts.
Whoever thought of streaming over UDP without an option for deeper
buffering clearly doesn't use maven.
What I am doing is moving my centos VM off the laptop and into rackspace
cloud. That saves RAM for the IDE, and as I'm testing things in the same
infrastructure, it gives me the ability to deploy artifacts at gigabit
rates. I just use git as a way of syncing source.
One thing I am debating -again on rackspace- is to set up Jenkins on
yet-another-VM, polling aggressively, and automatically running the full
test suite every half hour. That way, it does the full regression testing
on all changes on my branch, while I focus on the one or two tests that I
That's something I discussed a way back
-it's only now that I'm sitting down and really doing it -git & github
makes a difference as I can have my own personal branches for the CI
tooling to play with
Has anyone else tried anything like this
On 16 January 2013 00:50, Erik Paulson <[EMAIL PROTECTED]> wrote:
> Hello -
> I'm curious what Hadoop developers use for their day-to-day hacking on
> Hadoop. I'm talking changes to the Hadoop libraries and daemons, and not
> developing Map-Reduce jobs or using using the HDFS Client libraries to talk
> to a filesystem from an application.
> I've checked out Hadoop, made minor changes and built it with Maven, and
> tracked down the resulting artifacts in a target/ directory that I could
> deploy. Is this typically how a cloudera/hortonworks/mapr/etc dev works, or
> are the IDEs more common?
> I realize this sort of sounds like a dumb question, but I'm mostly curious
> what I might be missing out on if I stay away from anything other than vim,
> and not being entirely sure where maven might be caching jars that it uses
> to build, and how careful I have to be to ensure that my changes wind up in
> the right places without having to do a clean build every time.