Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Building and Deploying MRv2


Copy link to this message
-
Re: Building and Deploying MRv2
Praveen Sripati 2011-06-16, 01:45
Thomas,

Thanks for the response.

I completed the following steps.

- got the code from MR-279 branch.
- installed ant/maven.
- Got the protobuf code, compiled and installed it.

Do I need the avro-maven-plugin? When I ran the below command got the
error that the pom file was not found. Where do I get the jar and the
pom files for the avro-maven-plugin? I was able to get the source code
for them, but not the binaries.

mvn install:install-file
-Dfile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.jar
-DpomFile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.pom

Regards,
Praveen

On Wednesday 15 June 2011 08:19 PM, Thomas Graves wrote:
>
>
> On 6/15/11 8:54 AM, "Praveen Sripati"<[EMAIL PROTECTED]>  wrote:
>
>> Hi,
>>
>> I am trying to build and deploy MRv2 and following the instructions in the
>> INSTALL file. The instructions have to be modified after the recent
>> re-organisation (HADOOP-7106)
>>
>>
> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTAL>
> L
>>
>> 1. Is the code under MR-279 branch not sufficient? Should I get the common
>> code from the yahoo-merge branch and the HDFS code from the HDFS-1052 branch
>> based on the instructions - what's the dependency?
>
> I'll update the INSTALL doc.
>
> You should be able to checkout just the MR-279 branch now.  It pulls in the
> yahoo-merge version of common and hdfs.  Todd set that up to work
> automatically when unsplitting, from his email:
>
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>    (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>
> Checkout code:
> svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/
>
> -install protobuf - see here for instructions on that, I'm actually using
> version  2.4.1:
> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/yarn
> /README
>
> Setup env:
> - export MAVEN_OPTS=-Xmx512m
> - export LD_LIBRARY_PATH=/usr/local/lib
>
> build common
> - cd common
> - ant veryclean mvn-install tar
>
> build hdfs
> - cd hdfs
> - ant veryclean mvn-install -Dresolvers=internal tar
>
> build yarn
> - cd mapreduce
> - mvn clean install assembly:assembly
>   # If tests are failing, add this option to skip them:
> -Dmaven.test.skip.exec=true
> - ant veryclean jar jar-test tar -Dresolvers=internal
>
> Let me know if you have any problems.
>
>
>>
>> 2. Should I get the rest of the code (excluding common and HDFS) from the
>> MR-279 branch?
>
> Yes, see above.
>>
>> 3. What is the yahoo-merge branch?
>   yahoo-merge branch was used when yahoo merged everything back to apache
> from its private branch.  I believe it should be same as trunk for common
> and hdfs at this point.
>
> Tom
>