Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # dev >> development environment for hadoop core


+
Erik Paulson 2013-01-15, 23:50
+
Todd Lipcon 2013-01-16, 01:44
+
Andy Isaacson 2013-01-16, 02:08
+
Surenkumar Nihalani 2013-01-16, 03:38
Copy link to this message
-
Re: development environment for hadoop core
My setup ( I work from home)

# OS/X laptop w/ 30" monitor
# FTTC broadband, 55Mbit/s down, 15+ up -it's the upload bandwidth that
really helps development: http://www.flickr.com/photos/steve_l/8050751551/
# IntelliJ IDEA IDE, settings edited for a 2GB Heap
# Maven on the command line for builds
# I run a "mvn install -DskipTests" every morning to ensure that apache's
own -SNAPSHOT artifacts aren't pulled in.
# CentOS 6.3 VM for doing the full binary build & test, making my own RPMs,
etc.
# coffee.

One thing that annoys me is that I've got an airplay-driven hifi set up,
and during builds there's enough CPU/RAM load that the music has dropouts.
Whoever thought of streaming over UDP without an option for deeper
buffering clearly doesn't use maven.

What I am doing is moving my centos VM off the laptop and into rackspace
cloud. That saves RAM for the IDE, and as I'm testing things in the same
infrastructure, it gives me the ability to deploy artifacts at gigabit
rates. I just use git as a way of syncing source.

One thing I am debating -again on rackspace- is to set up Jenkins on
yet-another-VM, polling aggressively, and automatically running the full
test suite every half hour. That way, it does the full regression testing
on all changes on my branch, while I focus on the one or two tests that I
care about.
That's something I discussed a way back
https://docs.google.com/document/d/16v4SFYC6WSB-Y-B0Uo3IEhEhEN9pPYrtvhtR8KW9ouI/edit

-it's only now that I'm sitting down and really doing it -git & github
makes a difference as I can have my own personal branches for the CI
tooling to play with

Has anyone else tried anything like this

On 16 January 2013 00:50, Erik Paulson <[EMAIL PROTECTED]> wrote:

> Hello -
>
> I'm curious what Hadoop developers use for their day-to-day hacking on
> Hadoop. I'm talking changes to the Hadoop libraries and daemons, and not
> developing Map-Reduce jobs or using using the HDFS Client libraries to talk
> to a filesystem from an application.
>
> I've checked out Hadoop, made minor changes and built it with Maven, and
> tracked down the resulting artifacts in a target/ directory that I could
> deploy. Is this typically how a cloudera/hortonworks/mapr/etc dev works, or
> are the IDEs more common?
>
> I realize this sort of sounds like a dumb question, but I'm mostly curious
> what I might be missing out on if I stay away from anything other than vim,
> and not being entirely sure where maven might be caching jars that it uses
> to build, and how careful I have to be to ensure that my changes wind up in
> the right places without having to do a clean build every time.
>
> Thanks!
>
> -Erik
>
+
Glen Mazza 2013-01-16, 13:31
+
Erik Paulson 2013-01-21, 16:36
+
Colin McCabe 2013-01-21, 18:31
+
Gopal Vijayaraghavan 2013-01-16, 14:17
+
Hitesh Shah 2013-01-16, 19:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB