Hi,

Are there any instructions to start the ResourceManager, NodeManager,
HistoryServer from Eclipse? I got the code from SVN, compiled it and ran the
sample program. The projects have also been exported in Eclipse. I want to
run the RM, NM and HS from Eclipse, so as to see the flow and fix any
problems in the code.

Thanks,
Praveen

Praveen,

I'm not aware of any devs using eclipse to run MRv2, I haven't tried it myself.

 OTOH, if you do get to we are happy to follow your lead, please feel free to provide us instructions and we can update the README... thanks.

Arun

On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:

> Hi,
>
> Are there any instructions to start the ResourceManager, NodeManager,
> HistoryServer from Eclipse? I got the code from SVN, compiled it and ran the
> sample program. The projects have also been exported in Eclipse. I want to
> run the RM, NM and HS from Eclipse, so as to see the flow and fix any
> problems in the code.
>
> Thanks,
> Praveen

Arun,

I did try it out for sometime, but didn't work. Need to get more into the
code to fix the problem.

It's real difficult to use some editor instead of an IDE to do the
development, bug fixing or get familiar with the code.

I was able to run the original MapReduce applications from within Eclipse.

I created a JIRA for the same.

https://issues.apache.org/jira/browse/MAPREDUCE-2798

Thanks,
Praveen

On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> Praveen,
>
> I'm not aware of any devs using eclipse to run MRv2, I haven't tried it
> myself.
>
>  OTOH, if you do get to we are happy to follow your lead, please feel free
> to provide us instructions and we can update the README... thanks.
>
> Arun
>
> On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
>
> > Hi,
> >
> > Are there any instructions to start the ResourceManager, NodeManager,
> > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran
> the
> > sample program. The projects have also been exported in Eclipse. I want
> to
> > run the RM, NM and HS from Eclipse, so as to see the flow and fix any
> > problems in the code.
> >
> > Thanks,
> > Praveen
>
>

Hi,

I am able to run the RM, NM and HS from Eclipse with a couple of settings
and run the org.apache.hadoop.examples.RandomWriter from command
promptusing the 'hadoop jar' command. Would be nice for easier
development and
debugging of these components.

But, when I run the org.apache.hadoop.examples.RandomWriter from Eclipse, I
get the below exceptions. I created a new Java project in Eclipse and copied
the RandomWriter.java to it. This exception is getting repeated map number
of times, looks like the error is thrown for every map launch.

Not sure what this error means, any help is appreciated.

2011-08-13 21:44:35,785 INFO  mapreduce.Job (Job.java:printTaskEvents(1213))
- Task Id : attempt_1313250752016_0003_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
    at
org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
    ... 8 more

Thanks,
Praveen

On Wed, Aug 10, 2011 at 10:56 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

> Praveen,
>
> I'm not aware of any devs using eclipse to run MRv2, I haven't tried it
> myself.
>
>  OTOH, if you do get to we are happy to follow your lead, please feel free
> to provide us instructions and we can update the README... thanks.
>
> Arun
>
> On Aug 9, 2011, at 7:39 PM, Praveen Sripati wrote:
>
> > Hi,
> >
> > Are there any instructions to start the ResourceManager, NodeManager,
> > HistoryServer from Eclipse? I got the code from SVN, compiled it and ran
> the
> > sample program. The projects have also been exported in Eclipse. I want
> to
> > run the RM, NM and HS from Eclipse, so as to see the flow and fix any
> > problems in the code.
> >
> > Thanks,
> > Praveen
>
>

Hi,

I did spend some time on trying to run RM, NM and HS from within Eclipse,
but got partial success. Here are the steps I followed (just in case someone
is interested)

- Got the code from SVN.

- Ran the ant and mvn command for compiling Hadoop Common, HDFS and
MapReduce.

- Ran the 'mvn eclipse:eclipse' and 'ant eclipse' for the projects.

- Included the following in /etc/environment (a reboot was required for it
to take effect)

HADOOP_MAPRED_HOME="/home/praveensripati/Hadoop/trunk/mapreduce"
HADOOP_COMMON_HOME="/home/praveensripati/Hadoop/trunk/hadoop-common"
HADOOP_HDFS_HOME="/home/praveensripati/Hadoop/trunk/hdfs"
YARN_HOME="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
HADOOP_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_CONF_DIR="/home/praveensripati/Hadoop/conf"
YARN_INSTALL="/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT"
JAVA_HOME="/usr/lib/jvm/java-6-sun"

- Imported the java projects in Eclipse.

- In the 'Java Build Path' removed the jar dependencies in the Libraries Tab
and included the corresponding projects in the Sources Tab. Made sure that
the projects got compiled properly.

- Started the resourcemanager, nodemanager and historyserver from command
prompt using the yarn-daemon.sh command. This dumps out the java command
with the arguments. Stopped the resourcemanager, nodemanager and
historyserver again from the command propmpt.

- Ran the ResourceManager, NodeManager and JobHistoryServer from Eclipse,
passing the parameters got from the previous step.

- Created a Java project with Eclipse and copied
org.apache.hadoop.examples.RandomWriter.java and ran with the following
parameters

    -Dmapreduce.job.user.name=praveensripati
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912
/home/praveensripati/Hadoop/trunk/hadoop-common/target/hadoop-common-0.23.0-SNAPSHOT-bin/bin/output

and got the following exceptions for each of the mapper invocation. Could
not figure out the cause of this exception.

2011-08-13 21:44:35,785 INFO  mapreduce.Job (Job.java:printTaskEvents(1213))
- Task Id : attempt_1313250752016_0003_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1179)
    at
org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:180)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:640)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.RandomWriter$RandomMapper
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:247)
    at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1125)
    at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1177)
    ... 8 more

- But was able to run the RandomWriter from command propmpt using the below
command

./hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.23.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output

Thanks,
Praveen

On Sat, Aug 13, 2011 at 10:40 PM, Praveen Sripati
<[EMAIL PROTECTED]>wrote: