Hello,

I need to build hadoop in Linux as I need to make some small changes in the
code, but I don't know what is the simplest way to build hadoop. I googled it
and so far I only found two places that tell how to build hadoop. One is
http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/.
I downloaded apache forrest, and do as it
ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
-Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
and get an error:
     [exec] BUILD FAILED
     [exec] /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
java.lang.NullPointerException
What does this error mean? it seems apache forrest is used to create hadoop
document and I just want to rebuild hadoop java code. Is there a way for me to
just rebuild java code? I ran "ant", it seems to work successfully, but I don't
know if it really compiled the code.

the other place I found is to show how to build hadoop with eclipse. I use
macbook and I have to ssh to linux boxes to work on hadoop, so it's not a very
good option even if it can really work.

Best,
Da

The Java5 dependency is about to go from Hadoop. See HADOOP-7072. I
will try to commit it first thing next year. So, wait a couple of days
and you'll be all right.

Happy New Year everyone!
On Thu, Dec 30, 2010 at 22:08, Da Zheng <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I need to build hadoop in Linux as I need to make some small changes in the
> code, but I don't know what is the simplest way to build hadoop. I googled it
> and so far I only found two places that tell how to build hadoop. One is
> http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/.
> I downloaded apache forrest, and do as it
> ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
> -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
> and get an error:
>     [exec] BUILD FAILED
>     [exec] /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
> java.lang.NullPointerException
> What does this error mean? it seems apache forrest is used to create hadoop
> document and I just want to rebuild hadoop java code. Is there a way for me to
> just rebuild java code? I ran "ant", it seems to work successfully, but I don't
> know if it really compiled the code.
>
> the other place I found is to show how to build hadoop with eclipse. I use
> macbook and I have to ssh to linux boxes to work on hadoop, so it's not a very
> good option even if it can really work.
>
> Best,
> Da
>

Happy new year!

Thanks. After applying the patch, I can compile the code with
ant -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
but there is still something unclear to me:
first, how do I compile only the java code? it takes quite a while to rebuild
hadoop with the command above.
secondly, after I rebuild hadoop, I get some jar files such as
hadoop-0.20.3-dev-core.jar under the directory of build/. How do I use them? do
I need to move to some specific directory?
thirdly, what arguments I can use in ant?

Best,
Da

On 12/31/10 1:36 AM, Konstantin Boudnik wrote:
> The Java5 dependency is about to go from Hadoop. See HADOOP-7072. I
> will try to commit it first thing next year. So, wait a couple of days
> and you'll be all right.
>
> Happy New Year everyone!
>
>
> On Thu, Dec 30, 2010 at 22:08, Da Zheng <[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>> I need to build hadoop in Linux as I need to make some small changes in the
>> code, but I don't know what is the simplest way to build hadoop. I googled it
>> and so far I only found two places that tell how to build hadoop. One is
>> http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/.
>> I downloaded apache forrest, and do as it
>> ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
>> -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
>> and get an error:
>>     [exec] BUILD FAILED
>>     [exec] /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
>> java.lang.NullPointerException
>> What does this error mean? it seems apache forrest is used to create hadoop
>> document and I just want to rebuild hadoop java code. Is there a way for me to
>> just rebuild java code? I ran "ant", it seems to work successfully, but I don't
>> know if it really compiled the code.
>>
>> the other place I found is to show how to build hadoop with eclipse. I use
>> macbook and I have to ssh to linux boxes to work on hadoop, so it's not a very
>> good option even if it can really work.
>>
>> Best,
>> Da
>>

For question #3, this should be helpful:
http://ant.apache.org/manual/tasksoverview.html#compile

On Sat, Jan 1, 2011 at 6:53 AM, Da Zheng <[EMAIL PROTECTED]> wrote:

> Happy new year!
>
> Thanks. After applying the patch, I can compile the code with
> ant -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
> but there is still something unclear to me:
> first, how do I compile only the java code? it takes quite a while to
> rebuild
> hadoop with the command above.
> secondly, after I rebuild hadoop, I get some jar files such as
> hadoop-0.20.3-dev-core.jar under the directory of build/. How do I use
> them? do
> I need to move to some specific directory?
> thirdly, what arguments I can use in ant?
>
> Best,
> Da
>
> On 12/31/10 1:36 AM, Konstantin Boudnik wrote:
> > The Java5 dependency is about to go from Hadoop. See HADOOP-7072. I
> > will try to commit it first thing next year. So, wait a couple of days
> > and you'll be all right.
> >
> > Happy New Year everyone!
> >
> >
> > On Thu, Dec 30, 2010 at 22:08, Da Zheng <[EMAIL PROTECTED]> wrote:
> >> Hello,
> >>
> >> I need to build hadoop in Linux as I need to make some small changes in
> the
> >> code, but I don't know what is the simplest way to build hadoop. I
> googled it
> >> and so far I only found two places that tell how to build hadoop. One is
> >>
> http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/
> .
> >> I downloaded apache forrest, and do as it
> >> ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
> >> -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
> >> and get an error:
> >>     [exec] BUILD FAILED
> >>     [exec]
> /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
> >> java.lang.NullPointerException
> >> What does this error mean? it seems apache forrest is used to create
> hadoop
> >> document and I just want to rebuild hadoop java code. Is there a way for
> me to
> >> just rebuild java code? I ran "ant", it seems to work successfully, but
> I don't
> >> know if it really compiled the code.
> >>
> >> the other place I found is to show how to build hadoop with eclipse. I
> use
> >> macbook and I have to ssh to linux boxes to work on hadoop, so it's not
> a very
> >> good option even if it can really work.
> >>
> >> Best,
> >> Da
> >>
>
>

Sorry, I didn't express my question 3 clearly.
I suppose ant is very much like Makefile so there are some targets in build.xml
and we can use them to build hadoop in a certain way. It's actually related to
my first question. For example, if I want to just compile the code (java and the
related native C code) but not build documents, which target should I choose?

Best,
Da

On 1/1/11 10:31 AM, Ted Yu wrote:
> For question #3, this should be helpful:
> http://ant.apache.org/manual/tasksoverview.html#compile
>
> On Sat, Jan 1, 2011 at 6:53 AM, Da Zheng <[EMAIL PROTECTED]> wrote:
>
>> Happy new year!
>>
>> Thanks. After applying the patch, I can compile the code with
>> ant -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
>> but there is still something unclear to me:
>> first, how do I compile only the java code? it takes quite a while to
>> rebuild
>> hadoop with the command above.
>> secondly, after I rebuild hadoop, I get some jar files such as
>> hadoop-0.20.3-dev-core.jar under the directory of build/. How do I use
>> them? do
>> I need to move to some specific directory?
>> thirdly, what arguments I can use in ant?
>>
>> Best,
>> Da
>>
>> On 12/31/10 1:36 AM, Konstantin Boudnik wrote:
>>> The Java5 dependency is about to go from Hadoop. See HADOOP-7072. I
>>> will try to commit it first thing next year. So, wait a couple of days
>>> and you'll be all right.
>>>
>>> Happy New Year everyone!
>>>
>>>
>>> On Thu, Dec 30, 2010 at 22:08, Da Zheng <[EMAIL PROTECTED]> wrote:
>>>> Hello,
>>>>
>>>> I need to build hadoop in Linux as I need to make some small changes in
>> the
>>>> code, but I don't know what is the simplest way to build hadoop. I
>> googled it
>>>> and so far I only found two places that tell how to build hadoop. One is
>>>>
>> http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/
>> .
>>>> I downloaded apache forrest, and do as it
>>>> ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
>>>> -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
>>>> and get an error:
>>>>     [exec] BUILD FAILED
>>>>     [exec]
>> /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
>>>> java.lang.NullPointerException
>>>> What does this error mean? it seems apache forrest is used to create
>> hadoop
>>>> document and I just want to rebuild hadoop java code. Is there a way for
>> me to
>>>> just rebuild java code? I ran "ant", it seems to work successfully, but
>> I don't
>>>> know if it really compiled the code.
>>>>
>>>> the other place I found is to show how to build hadoop with eclipse. I
>> use
>>>> macbook and I have to ssh to linux boxes to work on hadoop, so it's not
>> a very
>>>> good option even if it can really work.
>>>>
>>>> Best,
>>>> Da
>>>>
>>
>>
>

How about compile-core-native ?

On Sat, Jan 1, 2011 at 11:21 AM, Da Zheng <[EMAIL PROTECTED]> wrote:

> Sorry, I didn't express my question 3 clearly.
> I suppose ant is very much like Makefile so there are some targets in
> build.xml
> and we can use them to build hadoop in a certain way. It's actually related
> to
> my first question. For example, if I want to just compile the code (java
> and the
> related native C code) but not build documents, which target should I
> choose?
>
> Best,
> Da
>
> On 1/1/11 10:31 AM, Ted Yu wrote:
> > For question #3, this should be helpful:
> > http://ant.apache.org/manual/tasksoverview.html#compile
> >
> > On Sat, Jan 1, 2011 at 6:53 AM, Da Zheng <[EMAIL PROTECTED]> wrote:
> >
> >> Happy new year!
> >>
> >> Thanks. After applying the patch, I can compile the code with
> >> ant -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
> >> but there is still something unclear to me:
> >> first, how do I compile only the java code? it takes quite a while to
> >> rebuild
> >> hadoop with the command above.
> >> secondly, after I rebuild hadoop, I get some jar files such as
> >> hadoop-0.20.3-dev-core.jar under the directory of build/. How do I use
> >> them? do
> >> I need to move to some specific directory?
> >> thirdly, what arguments I can use in ant?
> >>
> >> Best,
> >> Da
> >>
> >> On 12/31/10 1:36 AM, Konstantin Boudnik wrote:
> >>> The Java5 dependency is about to go from Hadoop. See HADOOP-7072. I
> >>> will try to commit it first thing next year. So, wait a couple of days
> >>> and you'll be all right.
> >>>
> >>> Happy New Year everyone!
> >>>
> >>>
> >>> On Thu, Dec 30, 2010 at 22:08, Da Zheng <[EMAIL PROTECTED]> wrote:
> >>>> Hello,
> >>>>
> >>>> I need to build hadoop in Linux as I need to make some small changes
> in
> >> the
> >>>> code, but I don't know what is the simplest way to build hadoop. I
> >> googled it
> >>>> and so far I only found two places that tell how to build hadoop. One
> is
> >>>>
> >>
> http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/
> >> .
> >>>> I downloaded apache forrest, and do as it
> >>>> ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
> >>>> -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
> >>>> and get an error:
> >>>>     [exec] BUILD FAILED
> >>>>     [exec]
> >> /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
> >>>> java.lang.NullPointerException
> >>>> What does this error mean? it seems apache forrest is used to create
> >> hadoop
> >>>> document and I just want to rebuild hadoop java code. Is there a way
> for
> >> me to
> >>>> just rebuild java code? I ran "ant", it seems to work successfully,
> but
> >> I don't
> >>>> know if it really compiled the code.
> >>>>
> >>>> the other place I found is to show how to build hadoop with eclipse. I
> >> use
> >>>> macbook and I have to ssh to linux boxes to work on hadoop, so it's
> not
> >> a very
> >>>> good option even if it can really work.
> >>>>
> >>>> Best,
> >>>> Da
> >>>>
> >>
> >>
> >
>
>

I think I get some ideas how to use different targets after using them.
Thanks,
Da

On 01/01/2011 02:26 PM, Ted Yu wrote:
> How about compile-core-native ?
>
> On Sat, Jan 1, 2011 at 11:21 AM, Da Zheng<[EMAIL PROTECTED]>  wrote:
>
>> Sorry, I didn't express my question 3 clearly.
>> I suppose ant is very much like Makefile so there are some targets in
>> build.xml
>> and we can use them to build hadoop in a certain way. It's actually related
>> to
>> my first question. For example, if I want to just compile the code (java
>> and the
>> related native C code) but not build documents, which target should I
>> choose?
>>
>> Best,
>> Da
>>
>> On 1/1/11 10:31 AM, Ted Yu wrote:
>>> For question #3, this should be helpful:
>>> http://ant.apache.org/manual/tasksoverview.html#compile
>>>
>>> On Sat, Jan 1, 2011 at 6:53 AM, Da Zheng<[EMAIL PROTECTED]>  wrote:
>>>
>>>> Happy new year!
>>>>
>>>> Thanks. After applying the patch, I can compile the code with
>>>> ant -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
>>>> but there is still something unclear to me:
>>>> first, how do I compile only the java code? it takes quite a while to
>>>> rebuild
>>>> hadoop with the command above.
>>>> secondly, after I rebuild hadoop, I get some jar files such as
>>>> hadoop-0.20.3-dev-core.jar under the directory of build/. How do I use
>>>> them? do
>>>> I need to move to some specific directory?
>>>> thirdly, what arguments I can use in ant?
>>>>
>>>> Best,
>>>> Da
>>>>
>>>> On 12/31/10 1:36 AM, Konstantin Boudnik wrote:
>>>>> The Java5 dependency is about to go from Hadoop. See HADOOP-7072. I
>>>>> will try to commit it first thing next year. So, wait a couple of days
>>>>> and you'll be all right.
>>>>>
>>>>> Happy New Year everyone!
>>>>>
>>>>>
>>>>> On Thu, Dec 30, 2010 at 22:08, Da Zheng<[EMAIL PROTECTED]>  wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I need to build hadoop in Linux as I need to make some small changes
>> in
>>>> the
>>>>>> code, but I don't know what is the simplest way to build hadoop. I
>>>> googled it
>>>>>> and so far I only found two places that tell how to build hadoop. One
>> is
>> http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/
>>>> .
>>>>>> I downloaded apache forrest, and do as it
>>>>>> ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
>>>>>> -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
>>>>>> and get an error:
>>>>>>      [exec] BUILD FAILED
>>>>>>      [exec]
>>>> /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
>>>>>> java.lang.NullPointerException
>>>>>> What does this error mean? it seems apache forrest is used to create
>>>> hadoop
>>>>>> document and I just want to rebuild hadoop java code. Is there a way
>> for
>>>> me to
>>>>>> just rebuild java code? I ran "ant", it seems to work successfully,
>> but
>>>> I don't
>>>>>> know if it really compiled the code.
>>>>>>
>>>>>> the other place I found is to show how to build hadoop with eclipse. I
>>>> use
>>>>>> macbook and I have to ssh to linux boxes to work on hadoop, so it's
>> not
>>>> a very
>>>>>> good option even if it can really work.
>>>>>>
>>>>>> Best,
>>>>>> Da
>>>>>>
>>>>
>>

On Fri, Dec 31, 2010 at 12:06 PM, Konstantin Boudnik <[EMAIL PROTECTED]> wrote:
> The Java5 dependency is about to go from Hadoop. See HADOOP-7072. I
> will try to commit it first thing next year. So, wait a couple of days
> and you'll be all right.

This is great news, as I'm suffering with an inability to run test-patch.

It always keeps failing with errors like:
docs:
     [exec] Apache Forrest.  Run 'forrest -projecthelp' to list options
     [exec]
     [exec] Error occurred during initialization of VM
     [exec] java/lang/NoClassDefFoundError: java/lang/Object

Would be great if this is committed soon. Thanks!

--
Harsh J
www.harshj.com

On Friday 31 December 2010 11:38 AM, Da Zheng wrote:
> I need to build hadoop in Linux as I need to make some small changes in the
> code, but I don't know what is the simplest way to build hadoop. I googled it
> and so far I only found two places that tell how to build hadoop. One is
> http://bigdata.wordpress.com/2010/05/27/hadoop-cookbook-3-how-to-build-your-own-hadoop-distribution/.
> I downloaded apache forrest, and do as it
> ant -Djava5.home=/usr/lib/jvm/java-1.5.0-gcj-4.4/
> -Dforrest.home=/home/zhengda/apache-forrest-0.8 compile-core tar
> and get an error:
>       [exec] BUILD FAILED
>       [exec] /home/zhengda/apache-forrest-0.8/main/targets/validate.xml:158:
> java.lang.NullPointerException

You're using GCJ instead of Sun's JDK. Try it with Sun's JDK 1.5:

   http://java.sun.com/products/archive/j2se/5.0_22/index.html
> What does this error mean? it seems apache forrest is used to create hadoop
> document and I just want to rebuild hadoop java code. Is there a way for me to
> just rebuild java code? I ran "ant", it seems to work successfully, but I don't
> know if it really compiled the code.

Don't specify the "tar" ant target then, though that does provide a
nicely-packaged tar-ball for deployment. You can check the ant messages
printed during the build process as well as the "build" sub-directory
to verify that the source-code has been built.

I personally use "ant clean binary" to get a suitable tar-ball. I have
a shell-script (called "bh" for "build hadoop") that invokes ant like
so:
-------------------------------- 8< --------------------------------
#!/bin/sh

TARGETS="clean binary"
if [ "$#" != 0 ]
then
   TARGETS="$*"
fi

export ANT_HOME="$HOME/apps/ant"
$ANT_HOME/bin/ant \
   -Dresolvers=internal \
   -Dcompile.native=true \
   -DONTcompile.c++=true \
   -Dxercescroot=/usr \
   -Djava5.home=$HOME/apps/jdk1_5 \
   -Dforrest.home=$HOME/apps/forrest \
   -Dfindbugs.home=$HOME/apps/findbugs \
   -Dscratch.dir=$HOME/src/Hadoop/tmp \
   -Dsvn.cmd=/usr/bin/svn \
   -Dgrep.cmd=/bin/grep \
   -Dpatch.cmd=/usr/bin/patch \
   $TARGETS
-------------------------------- 8< --------------------------------
(NOTE: The "-DONTcompile.c++=true" bit should ideally be
"-Dcompile.c++=true" but is that way till the issue in MAPREDUCE-2127
gets resolved.)

This script also incorporates the "test-patch.sh" defines suggested in
http://wiki.apache.org/hadoop/HowToContribute, which are needed only
if you're creating patches for Hadoop.

Of course, you need to have GNU make, binutils, GCC and Apache Xerces C
installed on your build-box, along with development packages like
"openssl-devel", "zlib-devel", etc., if you wish to compile native code.
(See http://wiki.apache.org/hadoop/NativeHadoop for details.)

On trunk, it is usually better to follow the sequence:

   1. (in "common") bh clean mvn-install
   2. (in "hdfs") bh veryclean mvn-install
   3. (in "mapreduce") bh veryclean mvn-install

after making sure that all Hadoop modules (common, hdfs & mapreduce)
are up-to-date with respect to the ASF subversion repository.

See also:

   http://wiki.apache.org/hadoop/BuildingHadoopFromSVN
   http://wiki.apache.org/hadoop/HowToRelease#Build_Requirements

and:

   https://github.com/omalley/hadoop-gpl-compression/wiki/FAQ

if you want to use GPL-ed compression libraries (LZO) with
Hadoop.

HTH,
Ranjit