Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> MRv2 Configuration in Eclipse


Copy link to this message
-
Re: MRv2 Configuration in Eclipse
Hi,

I did spend some time on trying to run RM, NM and HS from within Eclipse,
but got partial success. Here are the steps I followed (just in case someone
is interested)

- Got the code from SVN.

- Ran the ant and mvn command for compiling Hadoop Common, HDFS and
MapReduce.

- Ran the 'mvn eclipse:eclipse' and 'ant eclipse' for the projects.

- Included the following in /etc/environment (a reboot was required for it
to take effect)

HADOOP_MAPRED_HOME="/home/praveensripati/Hadoop/trunk/mapreduce"
HADOOP_COMMON_HOME="/home/praveensripati/Hadoop/trunk/hadoop-common"
HADOOP_HDFS_HOME="/home/praveensripati/Hadoop/trunk/hdfs"
YARN_HOME="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
HADOOP_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_INSTALL="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
JAVA_HOME="/usr/lib/jvm/java-6-sun"

- Imported the java projects in Eclipse.

- In the 'Java Build Path' removed the jar dependencies in the Libraries Tab
and included the corresponding projects in the Sources Tab. Made sure that
the projects got compiled properly.

- Started the resourcemanager, nodemanager and historyserver from command
prompt using the yarn-daemon.sh command. This dumps out the java command
with the arguments. Stopped the resourcemanager, nodemanager and
historyserver again from the command propmpt.

- Ran the ResourceManager, NodeManager and JobHistoryServer from Eclipse,
passing the parameters got from the previous step.

- Created a Java project with Eclipse and copied
org.apache.hadoop.examples.RandomWriter.java and ran with the following
parameters

    -Dmapreduce.job.user.name=praveensripati
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912
/home/praveensripati/Hadoop/trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin/output

and got the following exceptions for each of the mapper invocation. Could
not figure out the cause of this exception.

2011-08-13 21:44:35,785 INFO  mapreduce.Job (Job.java:printTaskEvents(1213))
- Task Id : attempt_1313250752016_0003_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
    at
org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
    ... 8 more

- But was able to run the RandomWriter from command propmpt using the below
command

./hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output

Thanks,
Praveen

On Sat, Aug 13, 2011 at 10:40 PM, Praveen Sripati
<[EMAIL PROTECTED]>wrote:

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB