Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # dev >> Development basis / rebuilding Cloudera dist


Copy link to this message
-
Re: Development basis / rebuilding Cloudera dist
On 22/10/11 06:30, Tim Broberg wrote:
> I'd like to add a core module to hadoop, but I'm running into some issues getting started.
>
> What I want is to be able to add a native library and codec to some stable build of hadoop, build, debug, experiment, and benchmark.
>
> Currently, I'm trying to rebuild the Cloudera rpms so I can get a complete stable set of source to start from. (When I tried working from the SVN trunk, it seemed there was so much active development going on, it was hard to get something stable to compile.)

try checking out
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security

for the 0.20.20x branch

https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23 for
the forthcoming 0.23 release. versions 0.23+ are built with Maven, which
makes some things easier (IDE setup), other things worse
>
> So, I'm working from the Cloudera instructions - https://ccp.cloudera.com/display/CDHDOC/Building+RPMs+from+CDH+Source+RPMs.
>
> I downloaded hadoop-0.20-0.20.2+923.97-1.src.rpm, installed jdk, ant, maven, and set various environment variables:
>      export PATH=$PATH:/usr/local/apache-maven-3.0.3/bin
>      export JAVA_HOME=/usr/java/jdk1.7.0
>      export HADOOP_HOME=/usr/lib/hadoop-0.20
>      export HADOOP_VERSION=`hadoop version | head -n 1 | cut -f 2 -d " "`
>
> I wasn't sure what to do about ANT_HOME, FORREST_HOME, or JAVA5_HOME as forrest wasn't requested to be installed, ant doesn't appear to have a special directory anywhere, and just generally not sure what's up with JAVA5_HOME, but this appears to be forrest related? None of these is generating complaints when I build.
>

ANT_HOME is the directory into which you installed Ant, see the ant
installation instructions

JAVA5_HOME is probably used by Forrest to create the documentation that
-if you don't ask for- won't need that or FORREST_HOME
> When I build, I get a whole bunch of output including the following:
>
> rpmbuild --rebuild $SRPM
>
> .........<snip>............
>
> compile:
>       [echo] contrib: gridmix
>      [javac] Compiling 31 source files to /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/build/contrib/gridmix/classes
>      [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:396: error: type argument ? extends T is not within bounds of type-variable E
>      [javac]   private<T>  String getEnumValues(Enum<? extends T>[] e) {
>      [javac]                                         ^
>      [javac]   where T,E are type-variables:
>      [javac]     T extends Object declared in method<T>getEnumValues(Enum<? extends T>[])
>      [javac]     E extends Enum<E>  declared in class Enum
>      [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:399: error: type argument ? extends T is not within bounds of type-variable E
>      [javac]     for (Enum<? extends T>  v : e) {
>      [javac]               ^
>      [javac]   where T,E are type-variables:
>      [javac]     T extends Object declared in method<T>getEnumValues(Enum<? extends T>[])
>      [javac]     E extends Enum<E>  declared in class Enum
>      [javac] Note: Some input files use unchecked or unsafe operations.
>      [javac] Note: Recompile with -Xlint:unchecked for details.
>      [javac] 2 errors

That could be a quirk of Java7 being fussier than Java6 -I've never seen
that error before. A quick search shows all mentions of it coming up
from July 2011

>
> Questions:
> 1 - Is there a more appropriate environment to work from than the Cloudera distribution for developing codecs?

check out the SVN source from svn.apache.org for the branch/version you
want to use

> 2 - If not, is this an ppropriate place to ask about how to build Cloudera?

I'd go to the cloudera lists

> 3 - Any suggestions for getting this rpm to rebuild?
> 4 - Any suggestions for editing the rpm so I can just wipe out gridmix altogether?

n/a