Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Re: built hadoop! please help with next steps?


Copy link to this message
-
RE: built hadoop! please help with next steps?
John Lilley 2013-06-04, 20:20
Answered my own question.  The Eclipse installs with Centos6 (or with yum) seems to have this problem.  A direct download of Eclipse for Java EE works fine.
John
From: John Lilley [mailto:[EMAIL PROTECTED]]
Sent: Monday, June 03, 2013 5:49 PM
To: [EMAIL PROTECTED]; Deepak Vohra
Subject: RE: built hadoop! please help with next steps?

I am getting errors trying to install m2e… has anyone else encountered this?
Cannot complete the install because one or more required items could not be found.
  Software being installed: m2e - Maven Integration for Eclipse 1.4.0.20130601-0314 (org.eclipse.m2e.feature.feature.group 1.4.0.20130601-0314)
  Missing requirement: Maven POM XML Editor 1.4.0.20130601-0314 (org.eclipse.m2e.editor.xml 1.4.0.20130601-0314) requires 'bundle org.eclipse.wst.xml.ui 0.0.0' but it could not be found
  Cannot satisfy dependency:
    From: m2e - Maven Integration for Eclipse 1.4.0.20130601-0314 (org.eclipse.m2e.feature.feature.group 1.4.0.20130601-0314)
    To: org.eclipse.m2e.editor.xml [1.4.0.20130601-0314]

From: Deepak Vohra [mailto:[EMAIL PROTECTED]]
Sent: Monday, June 03, 2013 4:12 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: built hadoop! please help with next steps?

John

The following patch is related to the issue cited.

https://issues.apache.org/jira/browse/HADOOP-9489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

thanks,
Deepak

________________________________
From: John Lilley <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Sent: Monday, June 3, 2013 1:51 PM
Subject: RE: built hadoop! please help with next steps?

I’ve followed the instructions in BUILDING.txt, generated the eclipse projects and imported the eclipse projects generated by maven using File -> Import -> General -> Existing project into workspace…
And they all appear.  However, the problems window shows:
Project 'hadoop-streaming' is missing required source folder: '/home/jlilley/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/conf'
Any idea what this means or how to fix it?
John

From: Sandy Ryza [mailto:[EMAIL PROTECTED]]
Sent: Friday, May 31, 2013 4:23 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: built hadoop! please help with next steps?

I've been successful with importing all the leaf-level maven projects as "Existing Maven Projects" using the eclipse maven plugin.  I've also gotten things to work without the eclipse maven plugin with some combination of mvn eclipse:eclipse, pointing to the m2repo, and the directory with the top pom.xml as my eclipse workspace directory.

On Fri, May 31, 2013 at 3:18 PM, John Lilley <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Sandy,
Thanks for all of the tips, I will try this over the weekend.   Regarding the last question, I am still trying to get the source loaded into Eclipse in a manner that facilitates easier browsing, symbol search, editing, etc.  Perhaps I am just missing some obvious FAQ?  This is leading up to modifying and debugging the “shell” ApplicationMaster sample.  This page:
http://stackoverflow.com/questions/11007423/developing-testing-and-debugging-hadoop-map-reduce-jobs-with-eclipse
looks promising as a Hadoop-in-Eclipse strategy, but it is over a year old and I’m not sure if it applies to Hadoop 2.0 and YARN.
John

From: Sandy Ryza [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Friday, May 31, 2013 12:13 PM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: built hadoop! please help with next steps?

Hi John,

Here's how I deploy/debug Hadoop locally:
To build and tar Hadoop:

  mvn clean package -Pdist -Dtar -DskipTests=true

The tar will be located in the project directory under hadoop-dist/target/.  I untar it into my deploy directory.

I then copy these scripts into the same directory:

hadoop-dev-env.sh:
#!/bin/bash
export HADOOP_DEV_HOME=`pwd`
export HADOOP_MAPRED_HOME=${HADOOP_DEV_HOME}
export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME}
export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME}
export YARN_HOME=${HADOOP_DEV_HOME}
export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop

hadoop-dev-setup.sh:
#!/bin/bash
source ./hadoop-dev-env.sh
bin/hadoop namenode -format

hadoop-dev.sh:
source hadoop-dev-env.sh
sbin/hadoop-daemon.sh $1 namenode
sbin/hadoop-daemon.sh $1 datanode
sbin/yarn-daemon.sh $1 resourcemanager
sbin/yarn-daemon.sh $1 nodemanager
sbin/mr-jobhistory-daemon.sh $1 historyserver
sbin/httpfs.sh $1

I copy all the files in <deploy directory>/conf into my conf directory, <deploy directory>/etc/hadoop, and then copy the minimal site configuration into .  The advantage of using a directory that's not the /conf directory is that it won't be overwritten the next time you untar a new build.  Lastly, I copy the minimal site configuration into the conf files.  For the sake of brevity, I won't include the properties in full xml format, but here are the ones I set:

yarn-site.xml:
  yarn.nodemanager.aux-services = mapreduce.shuffle
  yarn.nodemanager.aux-services.mapreduce.shuffle.class = org.apache.hadoop.mapred.ShuffleHandler
  yarn.resourcemanager.scheduler.class = org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
mapred-site.xml:
  mapreduce.framework.name<http://mapreduce.framework.name/> = yarn
core-site.xml:
  fs.default.name<http://fs.default.name/> = hdfs://localhost:9000
hdfs-site.xml:
  dfs.replication = 1
  dfs.permissions = false

Then, to format HDFS and start our cluster, we can simply do:
./hadoop-dev-setup.sh
./hadoop-dev.sh start
To stop it:
./hadoop-dev.sh stop

Once I have this set up, for quicker iteration, I have some scripts that build submodules (sometimes all of mapreduce, sometimes just the resourcemanager) and copy the updated jars into my setup.

Regarding your la