I'd like to add a core module to hadoop, but I'm running into some issues getting started.
 
What I want is to be able to add a native library and codec to some stable build of hadoop, build, debug, experiment, and benchmark.
 
Currently, I'm trying to rebuild the Cloudera rpms so I can get a complete stable set of source to start from. (When I tried working from the SVN trunk, it seemed there was so much active development going on, it was hard to get something stable to compile.)
 
So, I'm working from the Cloudera instructions - https://ccp.cloudera.com/display/CDHDOC/Building+RPMs+from+CDH+Source+RPMs.
 
I downloaded hadoop-0.20-0.20.2+923.97-1.src.rpm, installed jdk, ant, maven, and set various environment variables:
    export PATH=$PATH:/usr/local/apache-maven-3.0.3/bin
    export JAVA_HOME=/usr/java/jdk1.7.0
    export HADOOP_HOME=/usr/lib/hadoop-0.20
    export HADOOP_VERSION=`hadoop version | head -n 1 | cut -f 2 -d " "`
 
I wasn't sure what to do about ANT_HOME, FORREST_HOME, or JAVA5_HOME as forrest wasn't requested to be installed, ant doesn't appear to have a special directory anywhere, and just generally not sure what's up with JAVA5_HOME, but this appears to be forrest related? None of these is generating complaints when I build.
 
When I build, I get a whole bunch of output including the following:
 
rpmbuild --rebuild $SRPM
 
.........<snip>............
 
compile:
     [echo] contrib: gridmix
    [javac] Compiling 31 source files to /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/build/contrib/gridmix/classes
    [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:396: error: type argument ? extends T is not within bounds of type-variable E
    [javac]   private <T> String getEnumValues(Enum<? extends T>[] e) {
    [javac]                                         ^
    [javac]   where T,E are type-variables:
    [javac]     T extends Object declared in method <T>getEnumValues(Enum<? extends T>[])
    [javac]     E extends Enum<E> declared in class Enum
    [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:399: error: type argument ? extends T is not within bounds of type-variable E
    [javac]     for (Enum<? extends T> v : e) {
    [javac]               ^
    [javac]   where T,E are type-variables:
    [javac]     T extends Object declared in method <T>getEnumValues(Enum<? extends T>[])
    [javac]     E extends Enum<E> declared in class Enum
    [javac] Note: Some input files use unchecked or unsafe operations.
    [javac] Note: Recompile with -Xlint:unchecked for details.
    [javac] 2 errors

Questions:
1 - Is there a more appropriate environment to work from than the Cloudera distribution for developing codecs?
2 - If not, is this an appropriate place to ask about how to build Cloudera?
3 - Any suggestions for getting this rpm to rebuild?
4 - Any suggestions for editing the rpm so I can just wipe out gridmix altogether?
 
TIA
    - Tim.

Hey Tim,

+ cdh-user@ where someone can help with your specific issue. (bcc common-dev).

You may also want to check out Apache Bigtop:
http://incubator.apache.org/bigtop

Thanks,
Eli

On Fri, Oct 21, 2011 at 10:30 PM, Tim Broberg <[EMAIL PROTECTED]> wrote:
> I'd like to add a core module to hadoop, but I'm running into some issues getting started.
>
> What I want is to be able to add a native library and codec to some stable build of hadoop, build, debug, experiment, and benchmark.
>
> Currently, I'm trying to rebuild the Cloudera rpms so I can get a complete stable set of source to start from. (When I tried working from the SVN trunk, it seemed there was so much active development going on, it was hard to get something stable to compile.)
>
> So, I'm working from the Cloudera instructions - https://ccp.cloudera.com/display/CDHDOC/Building+RPMs+from+CDH+Source+RPMs.
>
> I downloaded hadoop-0.20-0.20.2+923.97-1.src.rpm, installed jdk, ant, maven, and set various environment variables:
>     export PATH=$PATH:/usr/local/apache-maven-3.0.3/bin
>     export JAVA_HOME=/usr/java/jdk1.7.0
>     export HADOOP_HOME=/usr/lib/hadoop-0.20
>     export HADOOP_VERSION=`hadoop version | head -n 1 | cut -f 2 -d " "`
>
> I wasn't sure what to do about ANT_HOME, FORREST_HOME, or JAVA5_HOME as forrest wasn't requested to be installed, ant doesn't appear to have a special directory anywhere, and just generally not sure what's up with JAVA5_HOME, but this appears to be forrest related? None of these is generating complaints when I build.
>
> When I build, I get a whole bunch of output including the following:
>
> rpmbuild --rebuild $SRPM
>
> .........<snip>............
>
> compile:
>      [echo] contrib: gridmix
>     [javac] Compiling 31 source files to /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/build/contrib/gridmix/classes
>     [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:396: error: type argument ? extends T is not within bounds of type-variable E
>     [javac]   private <T> String getEnumValues(Enum<? extends T>[] e) {
>     [javac]                                         ^
>     [javac]   where T,E are type-variables:
>     [javac]     T extends Object declared in method <T>getEnumValues(Enum<? extends T>[])
>     [javac]     E extends Enum<E> declared in class Enum
>     [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:399: error: type argument ? extends T is not within bounds of type-variable E
>     [javac]     for (Enum<? extends T> v : e) {
>     [javac]               ^
>     [javac]   where T,E are type-variables:
>     [javac]     T extends Object declared in method <T>getEnumValues(Enum<? extends T>[])
>     [javac]     E extends Enum<E> declared in class Enum
>     [javac] Note: Some input files use unchecked or unsafe operations.
>     [javac] Note: Recompile with -Xlint:unchecked for details.
>     [javac] 2 errors
>
> Questions:
> 1 - Is there a more appropriate environment to work from than the Cloudera distribution for developing codecs?
> 2 - If not, is this an appropriate place to ask about how to build Cloudera?
> 3 - Any suggestions for getting this rpm to rebuild?
> 4 - Any suggestions for editing the rpm so I can just wipe out gridmix altogether?
>
> TIA
>     - Tim.

On 22/10/11 06:30, Tim Broberg wrote:
> I'd like to add a core module to hadoop, but I'm running into some issues getting started.
>
> What I want is to be able to add a native library and codec to some stable build of hadoop, build, debug, experiment, and benchmark.
>
> Currently, I'm trying to rebuild the Cloudera rpms so I can get a complete stable set of source to start from. (When I tried working from the SVN trunk, it seemed there was so much active development going on, it was hard to get something stable to compile.)

try checking out
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security

for the 0.20.20x branch

https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23 for
the forthcoming 0.23 release. versions 0.23+ are built with Maven, which
makes some things easier (IDE setup), other things worse
>
> So, I'm working from the Cloudera instructions - https://ccp.cloudera.com/display/CDHDOC/Building+RPMs+from+CDH+Source+RPMs.
>
> I downloaded hadoop-0.20-0.20.2+923.97-1.src.rpm, installed jdk, ant, maven, and set various environment variables:
>      export PATH=$PATH:/usr/local/apache-maven-3.0.3/bin
>      export JAVA_HOME=/usr/java/jdk1.7.0
>      export HADOOP_HOME=/usr/lib/hadoop-0.20
>      export HADOOP_VERSION=`hadoop version | head -n 1 | cut -f 2 -d " "`
>
> I wasn't sure what to do about ANT_HOME, FORREST_HOME, or JAVA5_HOME as forrest wasn't requested to be installed, ant doesn't appear to have a special directory anywhere, and just generally not sure what's up with JAVA5_HOME, but this appears to be forrest related? None of these is generating complaints when I build.
>

ANT_HOME is the directory into which you installed Ant, see the ant
installation instructions

JAVA5_HOME is probably used by Forrest to create the documentation that
-if you don't ask for- won't need that or FORREST_HOME
> When I build, I get a whole bunch of output including the following:
>
> rpmbuild --rebuild $SRPM
>
> .........<snip>............
>
> compile:
>       [echo] contrib: gridmix
>      [javac] Compiling 31 source files to /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/build/contrib/gridmix/classes
>      [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:396: error: type argument ? extends T is not within bounds of type-variable E
>      [javac]   private<T>  String getEnumValues(Enum<? extends T>[] e) {
>      [javac]                                         ^
>      [javac]   where T,E are type-variables:
>      [javac]     T extends Object declared in method<T>getEnumValues(Enum<? extends T>[])
>      [javac]     E extends Enum<E>  declared in class Enum
>      [javac] /home/tbroberg/rpmbuild/BUILD/hadoop-0.20.2-cdh3u1/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java:399: error: type argument ? extends T is not within bounds of type-variable E
>      [javac]     for (Enum<? extends T>  v : e) {
>      [javac]               ^
>      [javac]   where T,E are type-variables:
>      [javac]     T extends Object declared in method<T>getEnumValues(Enum<? extends T>[])
>      [javac]     E extends Enum<E>  declared in class Enum
>      [javac] Note: Some input files use unchecked or unsafe operations.
>      [javac] Note: Recompile with -Xlint:unchecked for details.
>      [javac] 2 errors

That could be a quirk of Java7 being fussier than Java6 -I've never seen
that error before. A quick search shows all mentions of it coming up
from July 2011

>
> Questions:
> 1 - Is there a more appropriate environment to work from than the Cloudera distribution for developing codecs?

check out the SVN source from svn.apache.org for the branch/version you
want to use

> 2 - If not, is this an ppropriate place to ask about how to build Cloudera?

I'd go to the cloudera lists

> 3 - Any suggestions for getting this rpm to rebuild?
> 4 - Any suggestions for editing the rpm so I can just wipe out gridmix altogether?

n/a