Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # dev - Building and Deploying MRv2


Copy link to this message
-
Re: Building and Deploying MRv2
Praveen Sripati 2011-06-19, 06:06

Thanks to the group. Finally, I got the build and deploying going with
the following exception during the common build

      [exec] validate-sitemap:
      [exec]
/home/praveensripati/Installations/apache-forrest-0.8/main/webapp/resources/schema/relaxng/sitemap-v06.rng:72:31:
error: datatype library "http://www.w3.org/2001/XMLSchema-datatypes" not
recognized

      [exec] BUILD FAILED
      [exec]
/home/praveensripati/Installations/apache-forrest-0.8/main/targets/validate.xml:158:
Validation failed, messages should have been provided.

------

The INSTALL doc  
(http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL)
is incomplete. Here are the additional steps

- Dependency on the apache-forrest-0.8.tar.gz, apache-forrest-0.9.tar.gz
is causing ClassNotFound Exceptions for the Common build.

- The hadoop-mapred-examples-0.22.0-SNAPSHOT.jar is not there in the
$HADOOP_MAPRED_HOME/build. "ant examples -Dresolvers=internal" had to be
run.

- The configuration files in the below folder have to be copied to a
configuration folder and the same folder should be set to the
HADOOP_CONF_DIR variable and exported.

     - MR-279/common/conf
     - MR-279/hdfs/conf
     - MR-279/mapreduce/conf
     - $YARN_INSTALL/conf

- Had to install autoconf 'sudo apt-get install autoconf"on Ubuntu 11.04.

Thanks,
Praveen
On Saturday 18 June 2011 05:22 PM, Praveen Sripati wrote:
> Hi,
>
> I have got the code from the svn into into
> /home/praveensripati/Hadoop/ directory and untar'd the
> hadoop-mapreduce-1.0-SNAPSHOT-all.tar.gz file in the
> /home/praveensripati/Hadoop folder. The INSTALL document says to
> export the following variables. What should the HADOOP_CONF_DIR
> variable be set to - common or hdfs or mapreduce conf folder? The
> YARN_CONF_DIR is again pointing to the HADOOP_CONF_DIR.
>
> export HADOOP_MAPRED_HOME=/home/praveensripati/Hadoop/MR-279/mapreduce
> export HADOOP_COMMON_HOME=/home/praveensripati/Hadoop/MR-279/common
> export HADOOP_HDFS_HOME=/home/praveensripati/Hadoop/MR-279/hdfs
> export YARN_HOME=/home/praveensripati/Hadoop/hadoop-mapreduce-1.0-SNAPSHOT
> export HADOOP_CONF_DIR> export YARN_CONF_DIR=$HADOOP_CONF_DIR
> Thanks,
> Praveen
>
> On Saturday 18 June 2011 08:52 AM, Praveen Sripati wrote:
>>
>> Hi,
>>
>> Finally, got all the jars built. Now is the time to run the MRv2.
>>
>> It would be nice if the below documentation gets updated
>>
>> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
>>
>> ----------
>>
>> yarn build
>>
>> Ubuntu had an older version of the protoc binary, I did put the
>> latest protoc binary which I build in the PATH and the following
>> error is not coming
>>
>> yarn_protos.proto:4:8: Option "java_generate_equals_and_hash" unknown.
>>
>> Then had to install autoconf 'sudo apt-get install autoconf" and the
>> hadoop-mapreduce-1.0-SNAPSHOT-all.tar.gz file got generated.
>>
>> ----------
>>
>> common build
>>
>> After including the forrestor 8, the following error is not coming
>> (HADOOP-7394 has been created for the same)
>>
>>       [exec] Exception in thread "main"
>> java.lang.NoClassDefFoundError: org/apache/fop/messaging/MessageHandler
>>       [exec]     at
>> org.apache.cocoon.serialization.FOPSerializer.configure(FOPSerializer.java:122)
>>
>>
>> But, now getting the following error (the jars are getting generated
>> in the build folder)
>>
>>      [exec] validate-sitemap:
>>      [exec]
>> /home/praveensripati/Installations/apache-forrest-0.8/main/webapp/resources/schema/relaxng/sitemap-v06.rng:72:31:
>> error: datatype library "http://www.w3.org/2001/XMLSchema-datatypes"
>> not recognized
>>
>>      [exec] BUILD FAILED
>>      [exec]
>> /home/praveensripati/Installations/apache-forrest-0.8/main/targets/validate.xml:158:
>> Validation failed, messages should have been provided.
>> Thanks,
>> Praveen
>>
>> On Saturday 18 June 2011 03:44 AM, Siddharth Seth wrote:
>>> Ubuntu seems to install the protocol buffer library (protobuf-compiler) as