MapReduce, mail # dev - Re: trouble building MR-279 branch

RE: trouble building MR-279 branch

Ravi Teja 2011-08-05, 10:59

Hi,

The Hudson build is failing now with the following error.
I am also facing following error. How is this resolved, as this is the first
step of the build by which, even the yarn jars don't compile, neither the
jars are present in the repositories.

Even if we install the jars manually from the previous Hudson
build,it may fail the compilation if any code changes are done after it.

>From where are the jars taken, to resolve this dependency?

Please help me resolving this issue.
[ivy:resolve]
::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: UNRESOLVED
DEPENDENCIES ::
[ivy:resolve]
::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] ::
org.apache.hadoop#yarn-server-common;0.279.0: not found
[ivy:resolve] ::
org.apache.hadoop#hadoop-mapreduce-client-core;0.279.0: not found
[ivy:resolve] ::
org.apache.hadoop#yarn-common;0.279.0: not found
[ivy:resolve]
::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Thanks and Regards,
Ravi Teja
-----Original Message-----
From: Ravi Teja [mailto:[EMAIL PROTECTED]]
Sent: Thursday, July 28, 2011 5:09 PM
To: '[EMAIL PROTECTED]'
Subject: Re: trouble building MR-279 branch
Hi,

I have followed the procedure mentioned below, for building MRv2, but was
not able to succeed with the task. I have got some compilation errors.
Please help me resolving this issue.

[INFO]
------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] yarn-api
[INFO] yarn-common
[INFO] yarn-server-common
[INFO] yarn-server-nodemanager
[INFO] yarn-server-resourcemanager
[INFO] yarn-server-tests
[INFO] yarn-server
[INFO] yarn
[INFO] hadoop-mapreduce-client-core
[INFO] hadoop-mapreduce-client-common
[INFO] hadoop-mapreduce-client-shuffle
[INFO] hadoop-mapreduce-client-app
[INFO] hadoop-mapreduce-client-hs
[INFO] hadoop-mapreduce-client-jobclient
[INFO] hadoop-mapreduce-client
[INFO] hadoop-mapreduce
[INFO]

[INFO]
------------------------------------------------------------------------
[INFO] Building yarn-api 1.0-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run
(create-protobuf-generated-sources-directory) @ yarn-api ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO]
[INFO] --- exec-maven-plugin:1.2:exec (generate-sources) @ yarn-api ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ yarn-api
---
[INFO] Source directory:
D:\Code\MR-279\mapreduce\yarn\yarn-api\target\generated-sources\proto added.

[INFO]
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @
yarn-api ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
D:\Code\MR-279\mapreduce\yarn\yarn-api\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ yarn-api
---
[INFO] Compiling 121 source files to
D:\Code\MR-279\mapreduce\yarn\yarn-api\target\classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[22,33] package
org.apache.commons.logging does not exist
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[23,33] package
org.apache.commons.logging does not exist
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary[INFO]
[INFO] yarn-api .......................................... FAILURE [5.610s]
[INFO] yarn-common ....................................... SKIPPED
[INFO] yarn-server-common ................................ SKIPPED
[INFO] yarn-server-nodemanager ........................... SKIPPED
[INFO] yarn-server-resourcemanager ....................... SKIPPED
[INFO] yarn-server-tests ................................. SKIPPED
[INFO] yarn-server ....................................... SKIPPED
[INFO] yarn .............................................. SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 5.844s
[INFO] Finished at: Thu Jul 28 16:03:07 IST 2011
[INFO] Final Memory: 9M/35M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile
(default-compile) on project yarn-api: Compilation failure: Compilation
failure:
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[22,33] package
org.apache.commons.logging does not exist
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[23,33] package
org.apache.commons.logging does not exist
[ERROR] -> [Help 1]

Regards,
Ravi Teja
From: Josh Wills [mailto:[EMAIL PROTECTED]]
Sent: Saturday, July 02, 2011 6:34 AM
To: [EMAIL PROTECTED]
Subject: Re: trouble building MR-279 br

trouble building MR-279 branch

Jim Falgout 2011-07-01, 19:00

Hi,

I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?

The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.

Thanks,
Jim.

Re: trouble building MR-279 branch

Luke Lu 2011-07-01, 19:07

On Fri, Jul 1, 2011 at 12:00 PM, Jim Falgout <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?

https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL

> The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.

Tell us at which step you're having problems.

__Luke

RE: trouble building MR-279 branch

Anupam Seth 2011-07-01, 19:10

Hi Jim,

Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).

Build YARN
=========These are in addition to what you would normally do.

Install protobuf
use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked

install the protoc executable (configure, make, make install)
install the maven artifact (cd java; mvn install)

Installing protoc requires gcc 4.1.x or higher (RHEL5).

If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc

Setup environment

export MAVEN_OPTS=-Xmx512m
export LD_LIBRARY_PATH=/usr/local/lib

Checkout the MR-279 branch
svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279

Build common

cd MR-279/common
ant veryclean mvn-install tar

Build hdfs

cd ../hdfs
ant veryclean mvn-install -Dresolvers=internal tar

Build yarn

cd ../mapreduce
mvn clean install assembly:assembly
# If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true
# NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option
# (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
ant veryclean jar jar-test tar -Dresolvers=internal

-----Original Message-----
From: Jim Falgout [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 01, 2011 2:00 PM
To: [EMAIL PROTECTED]
Subject: trouble building MR-279 branch

Hi,

I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?

The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.

Thanks,
Jim.

RE: trouble building MR-279 branch

Jim Falgout 2011-07-01, 21:23

Thanks Anupam! I had skipped the step to get protobuf into the maven repository. Also, I had Apache Forrest version 0.9 installed. That caused the builds to fail on generating some docs. Installed version 0.8 and it worked smoothly.

On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?

-----Original Message-----
From: Anupam Seth [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 01, 2011 2:10 PM
To: [EMAIL PROTECTED]
Subject: RE: trouble building MR-279 branch

Hi Jim,

Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).

Build YARN
=========These are in addition to what you would normally do.

Install protobuf
use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked

install the protoc executable (configure, make, make install)
install the maven artifact (cd java; mvn install)

Installing protoc requires gcc 4.1.x or higher (RHEL5).

If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc

Setup environment

export MAVEN_OPTS=-Xmx512m
export LD_LIBRARY_PATH=/usr/local/lib

Checkout the MR-279 branch
svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279

Build common

cd MR-279/common
ant veryclean mvn-install tar

Build hdfs

cd ../hdfs
ant veryclean mvn-install -Dresolvers=internal tar

Build yarn

cd ../mapreduce
mvn clean install assembly:assembly
# If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
ant veryclean jar jar-test tar -Dresolvers=internal

-----Original Message-----
From: Jim Falgout [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 01, 2011 2:00 PM
To: [EMAIL PROTECTED]
Subject: trouble building MR-279 branch

Hi,

I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?

The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.

Thanks,
Jim.

RE: trouble building MR-279 branch

Anupam Seth 2011-07-01, 22:02

Hi Jim,

Not sure if this helps at all, but not sure if you could get an idea by looking at the included examples:

http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/src/examples/org/apache/hadoop/examples

that can be run as:

hadoop jar $HADOOP_MAPRED_HOME/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar wordcount input output

Regards,
Anupam

-----Original Message-----
From: Jim Falgout [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 01, 2011 4:23 PM
To: [EMAIL PROTECTED]
Subject: RE: trouble building MR-279 branch

Thanks Anupam! I had skipped the step to get protobuf into the maven repository. Also, I had Apache Forrest version 0.9 installed. That caused the builds to fail on generating some docs. Installed version 0.8 and it worked smoothly.

On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?

-----Original Message-----
From: Anupam Seth [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 01, 2011 2:10 PM
To: [EMAIL PROTECTED]
Subject: RE: trouble building MR-279 branch

Hi Jim,

Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).

Build YARN
=========These are in addition to what you would normally do.

Install protobuf
use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked

install the protoc executable (configure, make, make install)
install the maven artifact (cd java; mvn install)

Installing protoc requires gcc 4.1.x or higher (RHEL5).

If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc

Setup environment

export MAVEN_OPTS=-Xmx512m
export LD_LIBRARY_PATH=/usr/local/lib

Checkout the MR-279 branch
svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279

Build common

cd MR-279/common
ant veryclean mvn-install tar

Build hdfs

cd ../hdfs
ant veryclean mvn-install -Dresolvers=internal tar

Build yarn

cd ../mapreduce
mvn clean install assembly:assembly
# If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
ant veryclean jar jar-test tar -Dresolvers=internal

-----Original Message-----
From: Jim Falgout [mailto:[EMAIL PROTECTED]]
Sent: Friday, July 01, 2011 2:00 PM
To: [EMAIL PROTECTED]
Subject: trouble building MR-279 branch

Hi,

I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?

The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.

Thanks,
Jim.

Re: trouble building MR-279 branch

Josh Wills 2011-07-02, 01:03

Hey Anupam,

I think Jim is asking about building non-MR application masters using the
RM/NM. I think Arun has promised to post a design doc soon. In the meantime,
I'm just reading through the code and fixing little issues I come across
(e.g., https://issues.apache.org/jira/browse/MAPREDUCE-2630 and
https://issues.apache.org/jira/browse/MAPREDUCE-2633).

Jim, what sort of app are you looking to build? I'd be happy to have some
company wading through the code.

Best,
Josh

On Fri, Jul 1, 2011 at 3:02 PM, Anupam Seth <[EMAIL PROTECTED]> wrote:

> Hi Jim,
>
> Not sure if this helps at all, but not sure if you could get an idea by
> looking at the included examples:
>
>
> http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/src/examples/org/apache/hadoop/examples
>
> that can be run as:
>
> hadoop jar $HADOOP_MAPRED_HOME/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar
> wordcount input output
>
> Regards,
> Anupam
>
> -----Original Message-----
> From: Jim Falgout [mailto:[EMAIL PROTECTED]]
> Sent: Friday, July 01, 2011 4:23 PM
> To: [EMAIL PROTECTED]
> Subject: RE: trouble building MR-279 branch
>
> Thanks Anupam! I had skipped the step to get protobuf into the maven
> repository. Also, I had Apache Forrest version 0.9 installed. That caused
> the builds to fail on generating some docs. Installed version 0.8 and it
> worked smoothly.
>
> On another note: do you know of any design docs for building an application
> executor using the new resource manager? Just wondering how to get started?
>
> -----Original Message-----
> From: Anupam Seth [mailto:[EMAIL PROTECTED]]
> Sent: Friday, July 01, 2011 2:10 PM
> To: [EMAIL PROTECTED]
> Subject: RE: trouble building MR-279 branch
>
> Hi Jim,
>
> Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
>
> Build YARN
> =========> These are in addition to what you would normally do.
>
> Install protobuf
> use Protobuf 2.4.0a (Download from
> http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1
> and it worked
>
> install the protoc executable (configure, make, make install)
> install the maven artifact (cd java; mvn install)
>
> Installing protoc requires gcc 4.1.x or higher (RHEL5).
>
> If the make step fails with (Valid until a fix is released for protobuf
> 2.4.0a) ./google/protobuf/descriptor.h:1152: error:
> `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_'
> is private Replace descriptor.cc with
> http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
>
> Setup environment
>
> export MAVEN_OPTS=-Xmx512m
> export LD_LIBRARY_PATH=/usr/local/lib
>
> Checkout the MR-279 branch
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>
> Build common
>
> cd MR-279/common
> ant veryclean mvn-install tar
>
> Build hdfs
>
> cd ../hdfs
> ant veryclean mvn-install -Dresolvers=internal tar
>
> Build yarn
>
> cd ../mapreduce
> mvn clean install assembly:assembly
> # If tests are failing, add this option to skip them:
> -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must
> also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean
> install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
> ant veryclean jar jar-test tar -Dresolvers=internal
>
> -----Original Message-----
> From: Jim Falgout [mailto:[EMAIL PROTECTED]]
> Sent: Friday, July 01, 2011 2:00 PM
> To: [EMAIL PROTECTED]
> Subject: trouble building MR-279 branch
>
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to
> build. Any docs/blogs/notes around that may give some hints?
>
> The build is unable to resolve the yarn-server even though I ran a
> "mvn-install" in the yarn source area to install it in the local repository.
>
> Thanks,
> Jim.
>
>
>

Re: trouble building MR-279 branch

Arun C Murthy 2011-07-03, 19:16

Jim,

On Jul 1, 2011, at 2:23 PM, Jim Falgout wrote:

> On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?
>

Jim, we are in the process of doing a better job with docs etc.

Sharad just gave a talk on writing a new AM at the Hadoop Core Contributor Meeting on 7/1...

Sharad - could you upload your slides?

thanks,
Arun

> -----Original Message-----
> From: Anupam Seth [mailto:[EMAIL PROTECTED]]
> Sent: Friday, July 01, 2011 2:10 PM
> To: [EMAIL PROTECTED]
> Subject: RE: trouble building MR-279 branch
>
> Hi Jim,
>
> Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
>
> Build YARN
> =========> These are in addition to what you would normally do.
>
> Install protobuf
> use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked
>
> install the protoc executable (configure, make, make install)
> install the maven artifact (cd java; mvn install)
>
> Installing protoc requires gcc 4.1.x or higher (RHEL5).
>
> If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
>
> Setup environment
>
> export MAVEN_OPTS=-Xmx512m
> export LD_LIBRARY_PATH=/usr/local/lib
>
> Checkout the MR-279 branch
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>
> Build common
>
> cd MR-279/common
> ant veryclean mvn-install tar
>
> Build hdfs
>
> cd ../hdfs
> ant veryclean mvn-install -Dresolvers=internal tar
>
> Build yarn
>
> cd ../mapreduce
> mvn clean install assembly:assembly
> # If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
> ant veryclean jar jar-test tar -Dresolvers=internal
>
> -----Original Message-----
> From: Jim Falgout [mailto:[EMAIL PROTECTED]]
> Sent: Friday, July 01, 2011 2:00 PM
> To: [EMAIL PROTECTED]
> Subject: trouble building MR-279 branch
>
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
>
> The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
>
> Thanks,
> Jim.
>
>

Re: trouble building MR-279 branch

Jim Falgout 2011-07-03, 22:29

Thanks Arun, that's what I'm looking for. I hear you are working on a dev guide. I'm open to working with an early copy and providing feedback. Thanks.
----- Original Message -----
From: Arun C Murthy <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Cc: Sharad Agarwal <[EMAIL PROTECTED]>
Sent: Sun Jul 03 14:16:12 2011
Subject: Re: trouble building MR-279 branch

Jim,

On Jul 1, 2011, at 2:23 PM, Jim Falgout wrote:

> On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?
>

Jim, we are in the process of doing a better job with docs etc.

Sharad just gave a talk on writing a new AM at the Hadoop Core Contributor Meeting on 7/1...

Sharad - could you upload your slides?

thanks,
Arun

> -----Original Message-----
> From: Anupam Seth [mailto:[EMAIL PROTECTED]]
> Sent: Friday, July 01, 2011 2:10 PM
> To: [EMAIL PROTECTED]
> Subject: RE: trouble building MR-279 branch
>
> Hi Jim,
>
> Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
>
> Build YARN
> =========> These are in addition to what you would normally do.
>
> Install protobuf
> use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked
>
> install the protoc executable (configure, make, make install)
> install the maven artifact (cd java; mvn install)
>
> Installing protoc requires gcc 4.1.x or higher (RHEL5).
>
> If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
>
> Setup environment
>
> export MAVEN_OPTS=-Xmx512m
> export LD_LIBRARY_PATH=/usr/local/lib
>
> Checkout the MR-279 branch
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>
> Build common
>
> cd MR-279/common
> ant veryclean mvn-install tar
>
> Build hdfs
>
> cd ../hdfs
> ant veryclean mvn-install -Dresolvers=internal tar
>
> Build yarn
>
> cd ../mapreduce
> mvn clean install assembly:assembly
> # If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
> ant veryclean jar jar-test tar -Dresolvers=internal
>
> -----Original Message-----
> From: Jim Falgout [mailto:[EMAIL PROTECTED]]
> Sent: Friday, July 01, 2011 2:00 PM
> To: [EMAIL PROTECTED]
> Subject: trouble building MR-279 branch
>
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
>
> The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
>
> Thanks,
> Jim.
>
>

Re: trouble building MR-279 branch

Ravi Teja 2011-07-28, 11:38

Hi,

I have followed the procedure mentioned below, for building MRv2, but was
not able to succeed with the task. I have got some compilation errors.
Please help me resolving this issue.

[INFO]
------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] yarn-api
[INFO] yarn-common
[INFO] yarn-server-common
[INFO] yarn-server-nodemanager
[INFO] yarn-server-resourcemanager
[INFO] yarn-server-tests
[INFO] yarn-server
[INFO] yarn
[INFO] hadoop-mapreduce-client-core
[INFO] hadoop-mapreduce-client-common
[INFO] hadoop-mapreduce-client-shuffle
[INFO] hadoop-mapreduce-client-app
[INFO] hadoop-mapreduce-client-hs
[INFO] hadoop-mapreduce-client-jobclient
[INFO] hadoop-mapreduce-client
[INFO] hadoop-mapreduce
[INFO]

[INFO]
------------------------------------------------------------------------
[INFO] Building yarn-api 1.0-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run
(create-protobuf-generated-sources-directory) @ yarn-api ---
[INFO] Executing tasks

main:
[INFO] Executed tasks
[INFO]
[INFO] --- exec-maven-plugin:1.2:exec (generate-sources) @ yarn-api ---
[INFO]
[INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ yarn-api
---
[INFO] Source directory:
D:\Code\MR-279\mapreduce\yarn\yarn-api\target\generated-sources\proto added.

[INFO]
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @
yarn-api ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
D:\Code\MR-279\mapreduce\yarn\yarn-api\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ yarn-api
---
[INFO] Compiling 121 source files to
D:\Code\MR-279\mapreduce\yarn\yarn-api\target\classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[22,33] package
org.apache.commons.logging does not exist
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[23,33] package
org.apache.commons.logging does not exist
[INFO] 2 errors
[INFO] -------------------------------------------------------------
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] yarn-api .......................................... FAILURE [5.610s]
[INFO] yarn-common ....................................... SKIPPED
[INFO] yarn-server-common ................................ SKIPPED
[INFO] yarn-server-nodemanager ........................... SKIPPED
[INFO] yarn-server-resourcemanager ....................... SKIPPED
[INFO] yarn-server-tests ................................. SKIPPED
[INFO] yarn-server ....................................... SKIPPED
[INFO] yarn .............................................. SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 5.844s
[INFO] Finished at: Thu Jul 28 16:03:07 IST 2011
[INFO] Final Memory: 9M/35M
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile
(default-compile) on project yarn-api: Compilation failure: Compilation
failure:
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[22,33] package
org.apache.commons.logging does not exist
[ERROR]
\Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap
i\records\impl\pb\ContainerPBImpl.java:[23,33] package
org.apache.commons.logging does not exist
[ERROR] -> [Help 1]

Regards,
Ravi Teja
From: Josh Wills [mailto:[EMAIL PROTECTED]]
Sent: Saturday, July 02, 2011 6:34 AM
To: [EMAIL PROTECTED]
Subject: Re: trouble building MR-279 branch

Hey Anupam,

I think Jim is asking about building non-MR application masters using the
RM/NM. I think Arun has promised to post a design doc soon. In the meantime,
I'm just reading through the code and fixing little issues I come across
(e.g., https://issues.apache.org/jira/browse/MAPREDUCE-2630 and
https://issues.apache.org/jira/browse/MAPREDUCE-2633).

Jim, what sort of app are you looking to build? I'd be happy to have some
company wading through the code.

Best,
Josh

On Fri, Jul 1, 2011 at 3:02 PM, Anupam Seth <[EMAIL PROTECTED]> wrote:

http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/src/exa
mples/org/apache/hadoop/examples
application
started?
`google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_'
http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/de
scriptor.cc
repository.