Karthiek C
2013-02-01, 00:14
Harsh J
2013-02-01, 00:24
Amir Sanjar
2013-02-01, 14:16
yiyu jia
2013-02-01, 15:14
Karthiek C
2013-02-01, 14:44
Amir Sanjar
2013-02-01, 15:24
Karthiek C
2013-02-01, 16:02
Erik Paulson
2013-02-04, 14:00
Karthiek C
2013-02-04, 15:45
-
Help to setup latest Hadoop source code on EclipseKarthiek C 2013-02-01, 00:14
I am trying to get the latest hadoop source code running in Eclipse but
facing lot of build errors due to maven plugin dependencies. All the
documentations I found online were for very old versions of hadoop. Is
there any other documentation used by developer community to get the code
up and running? Can someone please help me out?
PS: I am working on a project where we want to develop a custom scheduler
for hadoop.
Thanks,
Karthiek
facing lot of build errors due to maven plugin dependencies. All the
documentations I found online were for very old versions of hadoop. Is
there any other documentation used by developer community to get the code
up and running? Can someone please help me out?
PS: I am working on a project where we want to develop a custom scheduler
for hadoop.
Thanks,
Karthiek
+
Karthiek C 2013-02-01, 00:14
-
Re: Help to setup latest Hadoop source code on EclipseHarsh J 2013-02-01, 00:24
Hi,
You should be able to do this (assuming your dev env. has all
side-deps like protoc 2.4.x+, etc. installed):
$ mvn clean install -DskipTests
$ mvn eclipse:eclipse
Then import all the necessary projects in from Eclipse.
On Fri, Feb 1, 2013 at 5:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
> I am trying to get the latest hadoop source code running in Eclipse but
> facing lot of build errors due to maven plugin dependencies. All the
> documentations I found online were for very old versions of hadoop. Is
> there any other documentation used by developer community to get the code
> up and running? Can someone please help me out?
>
> PS: I am working on a project where we want to develop a custom scheduler
> for hadoop.
>
> Thanks,
> Karthiek
--
Harsh J
You should be able to do this (assuming your dev env. has all
side-deps like protoc 2.4.x+, etc. installed):
$ mvn clean install -DskipTests
$ mvn eclipse:eclipse
Then import all the necessary projects in from Eclipse.
On Fri, Feb 1, 2013 at 5:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
> I am trying to get the latest hadoop source code running in Eclipse but
> facing lot of build errors due to maven plugin dependencies. All the
> documentations I found online were for very old versions of hadoop. Is
> there any other documentation used by developer community to get the code
> up and running? Can someone please help me out?
>
> PS: I am working on a project where we want to develop a custom scheduler
> for hadoop.
>
> Thanks,
> Karthiek
--
Harsh J
+
Harsh J 2013-02-01, 00:24
-
Re: Help to setup latest Hadoop source code on EclipseAmir Sanjar 2013-02-01, 14:16
what version of hadoop are you using? I have been debugging hadoop 1.1.x
source code using eclipse everyday. If you are using similar version, I
might be able to help.
Best Regards
Amir Sanjar
System Management Architect
PowerLinux Open Source Hadoop development lead
IBM Senior Software Engineer
Phone# 512-286-8393
Fax# 512-838-8858
source code using eclipse everyday. If you are using similar version, I
might be able to help.
Best Regards
Amir Sanjar
System Management Architect
PowerLinux Open Source Hadoop development lead
IBM Senior Software Engineer
Phone# 512-286-8393
Fax# 512-838-8858
+
Amir Sanjar 2013-02-01, 14:16
-
Re: Help to setup latest Hadoop source code on Eclipseyiyu jia 2013-02-01, 15:14
I followed this link to setup my hadoop development eclipse env.
http://wiki.apache.org/hadoop/EclipseEnvironment
You may ran into some error messages when you run mvn command. but, it can
be easily fixed. However, i do not remember what is the errors. just be
patient and read the error message. you can fix it.
On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
http://wiki.apache.org/hadoop/EclipseEnvironment
You may ran into some error messages when you run mvn command. but, it can
be easily fixed. However, i do not remember what is the errors. just be
patient and read the error message. you can fix it.
On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
+
yiyu jia 2013-02-01, 15:14
-
Re: Help to setup latest Hadoop source code on EclipseKarthiek C 2013-02-01, 14:44
That's great! So far I tried using trunk code. But 1.1.x works for me. Can
you please help me out? My purpose is to debug through hadoop's default
scheduler code and schedulers like fair share scheduler.
And thanks for the advice Harsh. I could generate eclipse project files
using the maven commands you gave, but couldn't get the project running in
eclipse.
Thanks,
Karthiek
On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
you please help me out? My purpose is to debug through hadoop's default
scheduler code and schedulers like fair share scheduler.
And thanks for the advice Harsh. I could generate eclipse project files
using the maven commands you gave, but couldn't get the project running in
eclipse.
Thanks,
Karthiek
On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
+
Karthiek C 2013-02-01, 14:44
-
Re: Help to setup latest Hadoop source code on EclipseAmir Sanjar 2013-02-01, 15:24
1) how to setup your development environment, for example:
export PATH=$PATH:~/development/protobuf-2.4.1/src
export HADOOP_HOME=~/development/hadoop-common-1.1.1
export HADOOP_LOG_DIR=$HADOOP_HOME/logs
export ECLIPSE_HOME=~/development/eclipse
export FORREST_HOME=~/development/apache-forrest-0.8
export PATH=$PATH:$FORREST_HOME/bin
export JAVA_HOME=~/development/ibm-java-x86_64-60
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=$CLASSPATH:$JAVA_HOME/jre/lib:$ANT_HOME/lib:
export LD_LIBRARY_PATH=$HADOOP_HOME/build/native/Linux-amd64-64/lib:
umask 0022
cd $HADOOP_HOME
2) do a clean build and test from command line to make sure everything
works without eclipse, example
ant clean package -Djava5.home=/usr/java/jdk1.5.0_22
-Dforrest.home=/home/sanjar/development/apache-forrest-0.8
-Dcompile.native=true
3) build an eclipse project
ant eclipse
4) import the project in to eclipse
file->import->general->"existing project into workspace"
5) Change eclipse jre to point to your sdk if needed.
6) add ANT_HOME to your eclipse enviroment
7) build the project clean
8) if debugging junit testcases add below argument to your "VM Argument" in
your "debug configuration".
-Dhadoop.log.dir=build/test/logs -Dtest.build.data=build/test/data
-Dhadoop.log.file=hadoop.log -Xmx500m -XX:+DumpOnOutOfMemoryError
-Djava.net.preferIPv4Stack=true
I might have missed a step :) , let me know if it helped
Best Regards
Amir Sanjar
System Management Architect
PowerLinux Open Source Hadoop development lead
IBM Senior Software Engineer
Phone# 512-286-8393
Fax# 512-838-8858
-----Karthiek C <[EMAIL PROTECTED]> wrote: -----
To: [EMAIL PROTECTED]
From: Karthiek C <[EMAIL PROTECTED]>
Date: 02/01/2013 08:45AM
Subject: Re: Help to setup latest Hadoop source code on Eclipse
That's great! So far I tried using trunk code. But 1.1.x works for me. Can
you please help me out? My purpose is to debug through hadoop's default
scheduler code and schedulers like fair share scheduler.
And thanks for the advice Harsh. I could generate eclipse project files
using the maven commands you gave, but couldn't get the project running in
eclipse.
Thanks,
Karthiek
On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> what version of hadoop are you using? I have been debugging hadoop 1.1.x
> source code using eclipse everyday. If you are using similar version, I
> might be able to help.
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
+
Amir Sanjar 2013-02-01, 15:24
-
Re: Help to setup latest Hadoop source code on EclipseKarthiek C 2013-02-01, 16:02
Thanks a lot Amir! This is great! Will try it out and update the group.
Regards,
Karthiek
On Fri, Feb 1, 2013 at 10:24 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
>
>
> 1) how to setup your development environment, for example:
> export PATH=$PATH:~/development/protobuf-2.4.1/src
> export HADOOP_HOME=~/development/hadoop-common-1.1.1
> export HADOOP_LOG_DIR=$HADOOP_HOME/logs
> export ECLIPSE_HOME=~/development/eclipse
> export FORREST_HOME=~/development/apache-forrest-0.8
> export PATH=$PATH:$FORREST_HOME/bin
> export JAVA_HOME=~/development/ibm-java-x86_64-60
> export PATH=$JAVA_HOME/bin:$PATH
> export CLASSPATH=$CLASSPATH:$JAVA_HOME/jre/lib:$ANT_HOME/lib:
> export LD_LIBRARY_PATH=$HADOOP_HOME/build/native/Linux-amd64-64/lib:
> umask 0022
> cd $HADOOP_HOME
>
> 2) do a clean build and test from command line to make sure everything
> works without eclipse, example
> ant clean package -Djava5.home=/usr/java/jdk1.5.0_22
> -Dforrest.home=/home/sanjar/development/apache-forrest-0.8
> -Dcompile.native=true
>
> 3) build an eclipse project
> ant eclipse
>
> 4) import the project in to eclipse
> file->import->general->"existing project into workspace"
>
> 5) Change eclipse jre to point to your sdk if needed.
> 6) add ANT_HOME to your eclipse enviroment
> 7) build the project clean
> 8) if debugging junit testcases add below argument to your "VM Argument" in
> your "debug configuration".
> -Dhadoop.log.dir=build/test/logs -Dtest.build.data=build/test/data
> -Dhadoop.log.file=hadoop.log -Xmx500m -XX:+DumpOnOutOfMemoryError
> -Djava.net.preferIPv4Stack=true
>
> I might have missed a step :) , let me know if it helped
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
> -----Karthiek C <[EMAIL PROTECTED]> wrote: -----
> To: [EMAIL PROTECTED]
> From: Karthiek C <[EMAIL PROTECTED]>
> Date: 02/01/2013 08:45AM
> Subject: Re: Help to setup latest Hadoop source code on Eclipse
>
> That's great! So far I tried using trunk code. But 1.1.x works for me. Can
> you please help me out? My purpose is to debug through hadoop's default
> scheduler code and schedulers like fair share scheduler.
>
> And thanks for the advice Harsh. I could generate eclipse project files
> using the maven commands you gave, but couldn't get the project running in
> eclipse.
>
> Thanks,
> Karthiek
>
>
> On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
> > what version of hadoop are you using? I have been debugging hadoop 1.1.x
> > source code using eclipse everyday. If you are using similar version, I
> > might be able to help.
> >
> > Best Regards
> > Amir Sanjar
> >
> > System Management Architect
> > PowerLinux Open Source Hadoop development lead
> > IBM Senior Software Engineer
> > Phone# 512-286-8393
> > Fax# 512-838-8858
> >
>
Regards,
Karthiek
On Fri, Feb 1, 2013 at 10:24 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
>
>
> 1) how to setup your development environment, for example:
> export PATH=$PATH:~/development/protobuf-2.4.1/src
> export HADOOP_HOME=~/development/hadoop-common-1.1.1
> export HADOOP_LOG_DIR=$HADOOP_HOME/logs
> export ECLIPSE_HOME=~/development/eclipse
> export FORREST_HOME=~/development/apache-forrest-0.8
> export PATH=$PATH:$FORREST_HOME/bin
> export JAVA_HOME=~/development/ibm-java-x86_64-60
> export PATH=$JAVA_HOME/bin:$PATH
> export CLASSPATH=$CLASSPATH:$JAVA_HOME/jre/lib:$ANT_HOME/lib:
> export LD_LIBRARY_PATH=$HADOOP_HOME/build/native/Linux-amd64-64/lib:
> umask 0022
> cd $HADOOP_HOME
>
> 2) do a clean build and test from command line to make sure everything
> works without eclipse, example
> ant clean package -Djava5.home=/usr/java/jdk1.5.0_22
> -Dforrest.home=/home/sanjar/development/apache-forrest-0.8
> -Dcompile.native=true
>
> 3) build an eclipse project
> ant eclipse
>
> 4) import the project in to eclipse
> file->import->general->"existing project into workspace"
>
> 5) Change eclipse jre to point to your sdk if needed.
> 6) add ANT_HOME to your eclipse enviroment
> 7) build the project clean
> 8) if debugging junit testcases add below argument to your "VM Argument" in
> your "debug configuration".
> -Dhadoop.log.dir=build/test/logs -Dtest.build.data=build/test/data
> -Dhadoop.log.file=hadoop.log -Xmx500m -XX:+DumpOnOutOfMemoryError
> -Djava.net.preferIPv4Stack=true
>
> I might have missed a step :) , let me know if it helped
>
> Best Regards
> Amir Sanjar
>
> System Management Architect
> PowerLinux Open Source Hadoop development lead
> IBM Senior Software Engineer
> Phone# 512-286-8393
> Fax# 512-838-8858
>
> -----Karthiek C <[EMAIL PROTECTED]> wrote: -----
> To: [EMAIL PROTECTED]
> From: Karthiek C <[EMAIL PROTECTED]>
> Date: 02/01/2013 08:45AM
> Subject: Re: Help to setup latest Hadoop source code on Eclipse
>
> That's great! So far I tried using trunk code. But 1.1.x works for me. Can
> you please help me out? My purpose is to debug through hadoop's default
> scheduler code and schedulers like fair share scheduler.
>
> And thanks for the advice Harsh. I could generate eclipse project files
> using the maven commands you gave, but couldn't get the project running in
> eclipse.
>
> Thanks,
> Karthiek
>
>
> On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
> > what version of hadoop are you using? I have been debugging hadoop 1.1.x
> > source code using eclipse everyday. If you are using similar version, I
> > might be able to help.
> >
> > Best Regards
> > Amir Sanjar
> >
> > System Management Architect
> > PowerLinux Open Source Hadoop development lead
> > IBM Senior Software Engineer
> > Phone# 512-286-8393
> > Fax# 512-838-8858
> >
>
+
Karthiek C 2013-02-01, 16:02
-
Re: Help to setup latest Hadoop source code on EclipseErik Paulson 2013-02-04, 14:00
I found that I needed to add the M2 Repo to the Eclipse classpath settings
in order to get a successful build. This page details several different
ways to accomplish that:
http://www.mkyong.com/maven/how-to-configure-m2_repo-variable-in-eclipse-ide/
-Erik
On Fri, Feb 1, 2013 at 8:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
> That's great! So far I tried using trunk code. But 1.1.x works for me. Can
> you please help me out? My purpose is to debug through hadoop's default
> scheduler code and schedulers like fair share scheduler.
>
> And thanks for the advice Harsh. I could generate eclipse project files
> using the maven commands you gave, but couldn't get the project running in
> eclipse.
>
> Thanks,
> Karthiek
>
>
> On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
> > what version of hadoop are you using? I have been debugging hadoop 1.1.x
> > source code using eclipse everyday. If you are using similar version, I
> > might be able to help.
> >
> > Best Regards
> > Amir Sanjar
> >
> > System Management Architect
> > PowerLinux Open Source Hadoop development lead
> > IBM Senior Software Engineer
> > Phone# 512-286-8393
> > Fax# 512-838-8858
> >
>
in order to get a successful build. This page details several different
ways to accomplish that:
http://www.mkyong.com/maven/how-to-configure-m2_repo-variable-in-eclipse-ide/
-Erik
On Fri, Feb 1, 2013 at 8:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
> That's great! So far I tried using trunk code. But 1.1.x works for me. Can
> you please help me out? My purpose is to debug through hadoop's default
> scheduler code and schedulers like fair share scheduler.
>
> And thanks for the advice Harsh. I could generate eclipse project files
> using the maven commands you gave, but couldn't get the project running in
> eclipse.
>
> Thanks,
> Karthiek
>
>
> On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
>
> > what version of hadoop are you using? I have been debugging hadoop 1.1.x
> > source code using eclipse everyday. If you are using similar version, I
> > might be able to help.
> >
> > Best Regards
> > Amir Sanjar
> >
> > System Management Architect
> > PowerLinux Open Source Hadoop development lead
> > IBM Senior Software Engineer
> > Phone# 512-286-8393
> > Fax# 512-838-8858
> >
>
+
Erik Paulson 2013-02-04, 14:00
-
Re: Help to setup latest Hadoop source code on EclipseKarthiek C 2013-02-04, 15:45
Thanks Erik. I am now trying to set up 1.1 version that uses ANT. Will use
your tip when I start working with latest version.
On Mon, Feb 4, 2013 at 9:00 AM, Erik Paulson <[EMAIL PROTECTED]> wrote:
> I found that I needed to add the M2 Repo to the Eclipse classpath settings
> in order to get a successful build. This page details several different
> ways to accomplish that:
>
>
> http://www.mkyong.com/maven/how-to-configure-m2_repo-variable-in-eclipse-ide/
>
> -Erik
>
>
> On Fri, Feb 1, 2013 at 8:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
>
> > That's great! So far I tried using trunk code. But 1.1.x works for me.
> Can
> > you please help me out? My purpose is to debug through hadoop's default
> > scheduler code and schedulers like fair share scheduler.
> >
> > And thanks for the advice Harsh. I could generate eclipse project files
> > using the maven commands you gave, but couldn't get the project running
> in
> > eclipse.
> >
> > Thanks,
> > Karthiek
> >
> >
> > On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> >
> > > what version of hadoop are you using? I have been debugging hadoop
> 1.1.x
> > > source code using eclipse everyday. If you are using similar version, I
> > > might be able to help.
> > >
> > > Best Regards
> > > Amir Sanjar
> > >
> > > System Management Architect
> > > PowerLinux Open Source Hadoop development lead
> > > IBM Senior Software Engineer
> > > Phone# 512-286-8393
> > > Fax# 512-838-8858
> > >
> >
>
your tip when I start working with latest version.
On Mon, Feb 4, 2013 at 9:00 AM, Erik Paulson <[EMAIL PROTECTED]> wrote:
> I found that I needed to add the M2 Repo to the Eclipse classpath settings
> in order to get a successful build. This page details several different
> ways to accomplish that:
>
>
> http://www.mkyong.com/maven/how-to-configure-m2_repo-variable-in-eclipse-ide/
>
> -Erik
>
>
> On Fri, Feb 1, 2013 at 8:44 AM, Karthiek C <[EMAIL PROTECTED]> wrote:
>
> > That's great! So far I tried using trunk code. But 1.1.x works for me.
> Can
> > you please help me out? My purpose is to debug through hadoop's default
> > scheduler code and schedulers like fair share scheduler.
> >
> > And thanks for the advice Harsh. I could generate eclipse project files
> > using the maven commands you gave, but couldn't get the project running
> in
> > eclipse.
> >
> > Thanks,
> > Karthiek
> >
> >
> > On Fri, Feb 1, 2013 at 9:16 AM, Amir Sanjar <[EMAIL PROTECTED]> wrote:
> >
> > > what version of hadoop are you using? I have been debugging hadoop
> 1.1.x
> > > source code using eclipse everyday. If you are using similar version, I
> > > might be able to help.
> > >
> > > Best Regards
> > > Amir Sanjar
> > >
> > > System Management Architect
> > > PowerLinux Open Source Hadoop development lead
> > > IBM Senior Software Engineer
> > > Phone# 512-286-8393
> > > Fax# 512-838-8858
> > >
> >
>
+
Karthiek C 2013-02-04, 15:45
