Here's how I deploy/debug Hadoop locally:
To build and tar Hadoop:
mvn clean package -Pdist -Dtar -DskipTests=true
The tar will be located in the project directory under hadoop-dist/target/.
I untar it into my deploy directory.
I then copy these scripts into the same directory:
bin/hadoop namenode -format
sbin/hadoop-daemon.sh $1 namenode
sbin/hadoop-daemon.sh $1 datanode
sbin/yarn-daemon.sh $1 resourcemanager
sbin/yarn-daemon.sh $1 nodemanager
sbin/mr-jobhistory-daemon.sh $1 historyserver
I copy all the files in <deploy directory>/conf into my conf directory,
<deploy directory>/etc/hadoop, and then copy the minimal site configuration
into . The advantage of using a directory that's not the /conf directory
is that it won't be overwritten the next time you untar a new build.
Lastly, I copy the minimal site configuration into the conf files. For
the sake of brevity, I won't include the properties in full xml format, but
here are the ones I set:
yarn.nodemanager.aux-services = mapreduce.shuffle
mapreduce.framework.name = yarn
fs.default.name = hdfs://localhost:9000
dfs.replication = 1
dfs.permissions = false
Then, to format HDFS and start our cluster, we can simply do:
To stop it:
Once I have this set up, for quicker iteration, I have some scripts that
build submodules (sometimes all of mapreduce, sometimes just the
resourcemanager) and copy the updated jars into my setup.
Regarding your last question, are you saying that you were able to load it
into Eclipse already, and want tips on the best way to browse within it?
Or that you're trying to get the source loaded into Eclipse?
Hope that helps!
On Thu, May 30, 2013 at 9:32 AM, John Lilley <[EMAIL PROTECTED]>wrote:
> Thanks for help me to build Hadoop! I’m through compile and install of
> maven plugins into Eclipse. I could use some pointers for next steps I
> want to take, which are:****
> **· **Deploy the simplest “development only” cluster (single
> node?) and learn how to debug within it. I read about the “local runner”
> configuration here (
> http://wiki.apache.org/hadoop/HowToDebugMapReducePrograms), does that
> still apply to MR2/YARN? It seems like an old page; perhaps there is a
> newer FAQ?****
> **· **Build and run the ApplicationMaster “shell” sample, and use
> that as a starting point for a customer AM. I would much appreciate any
> advice on getting the edit/build/debug cycle ironed out for an AM.****
> **· **Setup Hadoop source for easier browsing and learning
> (Eclipse load?). What is typically done to make for easy browsing of
> referenced classes/methods by name?****
> ** **
> ** **
John Lilley 2013-05-31, 22:18
Sandy Ryza 2013-05-31, 22:22
John Lilley 2013-06-04, 22:22
John Lilley 2013-06-04, 20:20