Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # dev >> Building and Deploying MRv2


Copy link to this message
-
Re: Building and Deploying MRv2

Thomas,

Thanks for the response.

I completed the following steps.

- got the code from MR-279 branch.
- installed ant/maven.
- Got the protobuf code, compiled and installed it.

Do I need the avro-maven-plugin? When I ran the below command got the
error that the pom file was not found. Where do I get the jar and the
pom files for the avro-maven-plugin? I was able to get the source code
for them, but not the binaries.

mvn install:install-file
-Dfile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.jar
-DpomFile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.pom

Regards,
Praveen

On Wednesday 15 June 2011 08:19 PM, Thomas Graves wrote:
>
>
> On 6/15/11 8:54 AM, "Praveen Sripati"<[EMAIL PROTECTED]>  wrote:
>
>> Hi,
>>
>> I am trying to build and deploy MRv2 and following the instructions in the
>> INSTALL file. The instructions have to be modified after the recent
>> re-organisation (HADOOP-7106)
>>
>>
> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTAL>
> L
>>
>> 1. Is the code under MR-279 branch not sufficient? Should I get the common
>> code from the yahoo-merge branch and the HDFS code from the HDFS-1052 branch
>> based on the instructions - what's the dependency?
>
> I'll update the INSTALL doc.
>
> You should be able to checkout just the MR-279 branch now.  It pulls in the
> yahoo-merge version of common and hdfs.  Todd set that up to work
> automatically when unsplitting, from his email:
>
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>    (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>
> Checkout code:
> svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/
>
> -install protobuf - see here for instructions on that, I'm actually using
> version  2.4.1:
> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/yarn
> /README
>
> Setup env:
> - export MAVEN_OPTS=-Xmx512m
> - export LD_LIBRARY_PATH=/usr/local/lib
>
> build common
> - cd common
> - ant veryclean mvn-install tar
>
> build hdfs
> - cd hdfs
> - ant veryclean mvn-install -Dresolvers=internal tar
>
> build yarn
> - cd mapreduce
> - mvn clean install assembly:assembly
>   # If tests are failing, add this option to skip them:
> -Dmaven.test.skip.exec=true
> - ant veryclean jar jar-test tar -Dresolvers=internal
>
> Let me know if you have any problems.
>
>
>>
>> 2. Should I get the rest of the code (excluding common and HDFS) from the
>> MR-279 branch?
>
> Yes, see above.
>>
>> 3. What is the yahoo-merge branch?
>   yahoo-merge branch was used when yahoo merged everything back to apache
> from its private branch.  I believe it should be same as trunk for common
> and hdfs at this point.
>
> Tom
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB