I am trying to get the latest hadoop source code running in Eclipse but
facing lot of build errors due to maven plugin dependencies. All the
documentations I found online were for very old versions of hadoop. Is
there any other documentation used by developer community to get the code
up and running? Can someone please help me out?

PS: I am working on a project where we want to develop a custom scheduler
for hadoop.

Thanks,
Karthiek

Hi,

You should be able to do this (assuming your dev env. has all
side-deps like protoc 2.4.x+, etc. installed):

$ mvn clean install -DskipTests
$ mvn eclipse:eclipse

Then import all the necessary projects in from Eclipse.

On Fri, Feb 1, 2013 at 5:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
> I am trying to get the latest hadoop source code running in Eclipse but
> facing lot of build errors due to maven plugin dependencies. All the
> documentations I found online were for very old versions of hadoop. Is
> there any other documentation used by developer community to get the code
> up and running? Can someone please help me out?
>
> PS: I am working on a project where we want to develop a custom scheduler
> for hadoop.
>
> Thanks,
> Karthiek

--
Harsh J

what version of hadoop are you using? I have been debugging hadoop 1.1.x
source code using eclipse everyday. If you are using similar version, I
might be able to help.

Best Regards
Amir Sanjar

System Management Architect
PowerLinux Open Source Hadoop development lead
IBM Senior Software Engineer
Phone# 512-286-8393
Fax#      512-838-8858

I followed this link to setup my hadoop development eclipse env.

http://wiki.apache.org/hadoop/EclipseEnvironment

You may ran into some error messages when you run mvn command. but, it can
be easily fixed. However, i do not remember what is the errors. just be
patient and read the error message. you can fix it.

On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:

> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax#      512-838-8858
>

That's great! So far I tried using trunk code. But 1.1.x works for me. Can
you please help me out? My purpose is to debug through hadoop's default
scheduler code and schedulers like fair share scheduler.

And thanks for the advice Harsh. I could generate eclipse project files
using the maven commands you gave, but couldn't get the project running in
eclipse.

Thanks,
Karthiek
On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:

> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax#      512-838-8858
>

1) how to setup your development environment, for example:
export PATH=$PATH:~/development/protobuf-2.4.1/src
export HADOOP_HOME=~/development/hadoop-common-1.1.1
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export ECLIPSE_HOME=~/development/eclipse
export FORREST_HOME=~/development/apache-forrest-0.8
export PATH=$PATH:$FORREST_HOME/bin
export JAVA_HOME=~/development/ibm-java-x86_64-60
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:$JAVA_HOME/jre/lib:$ANT_HOME/lib:
export LD_LIBRARY_PATH=$HADOOP_HOME/build/native/Linux-amd64-64/lib:
umask 0022
cd $HADOOP_HOME

2) do a clean build and test from command line to make sure everything
works without eclipse, example
ant clean package -Djava5.home=/usr/java/jdk1.5.0_22
-Dforrest.home=/home/sanjar/development/apache-forrest-0.8
-Dcompile.native=true

3) build an eclipse project
ant eclipse

4) import the project in to eclipse
file->import->general->"existing project into workspace"

5) Change eclipse jre to point to your sdk if needed.
6) add ANT_HOME to your eclipse enviroment
7) build the project clean
8) if debugging junit testcases add below argument to your "VM Argument" in
your "debug configuration".
-Dhadoop.log.dir=build/test/logs -Dtest.build.data=build/test/data
-Dhadoop.log.file=hadoop.log -Xmx500m -XX:+DumpOnOutOfMemoryError
-Djava.net.preferIPv4Stack=true

I might have missed a step :) , let me know if it helped

Best Regards
Amir Sanjar

System Management Architect
PowerLinux Open Source Hadoop development lead
IBM Senior Software Engineer
Phone# 512-286-8393
Fax# 512-838-8858

-----Karthiek C <[EMAIL PROTECTED]> wrote: -----
To: [EMAIL PROTECTED]
From: Karthiek C <[EMAIL PROTECTED]>
Date: 02/01/2013 08:45AM
Subject: Re: Help to setup latest Hadoop source code on Eclipse

That's great! So far I tried using trunk code. But 1.1.x works for me. Can
you please help me out? My purpose is to debug through hadoop's default
scheduler code and schedulers like fair share scheduler.

And thanks for the advice Harsh. I could generate eclipse project files
using the maven commands you gave, but couldn't get the project running in
eclipse.

Thanks,
Karthiek
On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:

> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax#      512-838-8858
>

Thanks a lot Amir! This is great! Will try it out and update the group.
Regards,
Karthiek
On Fri, Feb 1, 2013 at 10:24 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:

>
>
>
> 1) how to setup your development environment, for example:
> export PATH=$PATH:~/development/protobuf-2.4.1/src
> export HADOOP_HOME=~/development/hadoop-common-1.1.1
> export HADOOP_LOG_DIR=$HADOOP_HOME/logs
> export ECLIPSE_HOME=~/development/eclipse
> export FORREST_HOME=~/development/apache-forrest-0.8
> export PATH=$PATH:$FORREST_HOME/bin
> export JAVA_HOME=~/development/ibm-java-x86_64-60
> export PATH=$JAVA_HOME/bin:$PATH
> export CLASSPATH=$CLASSPATH:$JAVA_HOME/jre/lib:$ANT_HOME/lib:
> export LD_LIBRARY_PATH=$HADOOP_HOME/build/native/Linux-amd64-64/lib:
> umask 0022
> cd $HADOOP_HOME
>
> 2) do a clean build and test from command line to make sure everything
> works without eclipse, example
> ant clean package -Djava5.home=/usr/java/jdk1.5.0_22
> -Dforrest.home=/home/sanjar/development/apache-forrest-0.8
> -Dcompile.native=true
>
> 3) build an eclipse project
> ant eclipse
>
> 4) import the project in to eclipse
> file->import->general->"existing project into workspace"
>
> 5) Change eclipse jre to point to your sdk if needed.
> 6) add ANT_HOME to your eclipse enviroment
> 7) build the project clean
> 8) if debugging junit testcases add below argument to your "VM Argument" in
> your "debug configuration".
> -Dhadoop.log.dir=build/test/logs -Dtest.build.data=build/test/data
> -Dhadoop.log.file=hadoop.log -Xmx500m -XX:+DumpOnOutOfMemoryError
> -Djava.net.preferIPv4Stack=true
>
> I might have missed a step :) , let me know if it helped
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
> -----Karthiek C <[EMAIL PROTECTED]> wrote: -----
> To: [EMAIL PROTECTED]
> From: Karthiek C <[EMAIL PROTECTED]>
> Date: 02/01/2013 08:45AM
> Subject: Re: Help to setup latest Hadoop source code on Eclipse
>
> That's great! So far I tried using trunk code. But 1.1.x works for me. Can
> you please help me out? My purpose is to debug through hadoop's default
> scheduler code and schedulers like fair share scheduler.
>
> And thanks for the advice Harsh. I could generate eclipse project files
> using the maven commands you gave, but couldn't get the project running in
> eclipse.
>
> Thanks,
> Karthiek
>
>
> On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
> > what version of hadoop are you using? I have been debugging hadoop 1.1.x
> > source code using eclipse everyday. If you are using similar version, I
> > might be able to help.
> >
> > Best Regards
> > Amir Sanjar
> >
> > System Management Architect
> > PowerLinux Open Source Hadoop development lead
> > IBM Senior Software Engineer
> > Phone# 512-286-8393
> > Fax#      512-838-8858
> >
>

I found that I needed to add the M2 Repo to the Eclipse classpath settings
in order to get a successful build. This page details several different
ways to accomplish that:

http://www.mkyong.com/maven/how-to-configure-m2_repo-variable-in-eclipse-ide/

-Erik
On Fri, Feb 1, 2013 at 8:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:

> That's great! So far I tried using trunk code. But 1.1.x works for me. Can
> you please help me out? My purpose is to debug through hadoop's default
> scheduler code and schedulers like fair share scheduler.
>
> And thanks for the advice Harsh. I could generate eclipse project files
> using the maven commands you gave, but couldn't get the project running in
> eclipse.
>
> Thanks,
> Karthiek
>
>
> On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
> > what version of hadoop are you using? I have been debugging hadoop 1.1.x
> > source code using eclipse everyday. If you are using similar version, I
> > might be able to help.
> >
> > Best Regards
> > Amir Sanjar
> >
> > System Management Architect
> > PowerLinux Open Source Hadoop development lead
> > IBM Senior Software Engineer
> > Phone# 512-286-8393
> > Fax#      512-838-8858
> >
>

Thanks Erik. I am now trying to set up 1.1 version that uses ANT. Will use
your tip when I start working with latest version.
On Mon, Feb 4, 2013 at 9:00 AM, Erik Paulson <[EMAIL PROTECTED]> wrote:

> I found that I needed to add the M2 Repo to the Eclipse classpath settings
> in order to get a successful build. This page details several different
> ways to accomplish that:
>
>
> http://www.mkyong.com/maven/how-to-configure-m2_repo-variable-in-eclipse-ide/
>
> -Erik
>
>
> On Fri, Feb 1, 2013 at 8:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
>
> > That's great! So far I tried using trunk code. But 1.1.x works for me.
> Can
> > you please help me out? My purpose is to debug through hadoop's default
> > scheduler code and schedulers like fair share scheduler.
> >
> > And thanks for the advice Harsh. I could generate eclipse project files
> > using the maven commands you gave, but couldn't get the project running
> in
> > eclipse.
> >
> > Thanks,
> > Karthiek
> >
> >
> > On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> >
> > > what version of hadoop are you using? I have been debugging hadoop
> 1.1.x
> > > source code using eclipse everyday. If you are using similar version, I
> > > might be able to help.
> > >
> > > Best Regards
> > > Amir Sanjar
> > >
> > > System Management Architect
> > > PowerLinux Open Source Hadoop development lead
> > > IBM Senior Software Engineer
> > > Phone# 512-286-8393
> > > Fax#      512-838-8858
> > >
> >
>