Hi all,

We'd like to use CMake instead of autotools to build native (C/C++) code in
Hadoop.  There are a lot of reasons to want to do this.  For one thing, it is
not feasible to use autotools on the Windows platform, because it depends on
UNIX shell scripts, the m4 macro processor, and some other pieces of
infrastructure which are not present on Windows.

For another thing, CMake builds are substantially simpler and faster, because
there is only one layer of generated code.  With autotools, you have automake
generating m4 code which autoconf reads, which it uses to generate a UNIX shell
script, which then generates another UNIX shell script, which eventually
generates Makefiles.  CMake simply generates Makefiles out of CMakeLists.txt
files-- much simpler to understand and debug, and much faster.
CMake is a lot easier to learn.

automake error messages can be very, very confusing.  This is because you are
essentially debugging a pile of shell scripts and macros, rather than a
coherent whole.  So you see error messages like "autoreconf: cannot empty
/tmp/ar0.4849: Is a directory" or "Can't locate object method "path" via
package "Autom4te... and so forth.  CMake error messages come from the CMake
application and they almost always immediately point you to the problem.

>From a build point of view, the net result of adopting CMake would be that you
would no longer need automake and related programs installed to build the
native parts of Hadoop.  Instead, you would need CMake installed.  CMake is
packaged by Red Hat, even in RHEL5, so it shouldn't be difficult to install
locally.  It's also available for Mac OS X and Windows, as I mentioned earlier.

The JIRA for this work is at https://issues.apache.org/jira/browse/HADOOP-8368
Thanks for reading.

sincerely,
Colin

+1   having a build tool that supports multiple platforms is worth the
dependency. I've also had good experiences with cmake.
On Mon, May 21, 2012 at 6:00 PM, Colin McCabe <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> We'd like to use CMake instead of autotools to build native (C/C++) code in
> Hadoop.  There are a lot of reasons to want to do this.  For one thing, it is
> not feasible to use autotools on the Windows platform, because it depends on
> UNIX shell scripts, the m4 macro processor, and some other pieces of
> infrastructure which are not present on Windows.
>
> For another thing, CMake builds are substantially simpler and faster, because
> there is only one layer of generated code.  With autotools, you have automake
> generating m4 code which autoconf reads, which it uses to generate a UNIX shell
> script, which then generates another UNIX shell script, which eventually
> generates Makefiles.  CMake simply generates Makefiles out of CMakeLists.txt
> files-- much simpler to understand and debug, and much faster.
> CMake is a lot easier to learn.
>
> automake error messages can be very, very confusing.  This is because you are
> essentially debugging a pile of shell scripts and macros, rather than a
> coherent whole.  So you see error messages like "autoreconf: cannot empty
> /tmp/ar0.4849: Is a directory" or "Can't locate object method "path" via
> package "Autom4te... and so forth.  CMake error messages come from the CMake
> application and they almost always immediately point you to the problem.
>
> From a build point of view, the net result of adopting CMake would be that you
> would no longer need automake and related programs installed to build the
> native parts of Hadoop.  Instead, you would need CMake installed.  CMake is
> packaged by Red Hat, even in RHEL5, so it shouldn't be difficult to install
> locally.  It's also available for Mac OS X and Windows, as I mentioned earlier.
>
> The JIRA for this work is at https://issues.apache.org/jira/browse/HADOOP-8368
> Thanks for reading.
>
> sincerely,
> Colin

i have better experience with scons instead of cmake

On 22 May 2012 11:33, Radim Kolar <[EMAIL PROTECTED]> wrote:

> i have better experience with scons instead of cmake
>

mmm, may be better to jump beyond make altogether to the higher levels of
nativeish build tools.

What would be good is that the output can be parsed by jenkins & have that
set up to build and test on windows and linux (and maybe BSD, OS/X if they
can play with VMs)

After a quick look at cmake, it seems reasonable. Please use feature tests
rather than OS tests. (By that I mean that if Solaris needs "foobar.h" and
RHEL needs "sys/foobar.h" to get the definition for foobar, the code should
ifdef whether they need sys/foobar.h and not if it is solaris or not.)

On a related note, on what platforms does the native code currently compile
and work?

I've compiled and tested on:
  RHEL & CentOS 5
  RHEL & CentOS 6

I've heard of it compiling on:
  Solaris

What other ones are out there?

On Wed, May 23, 2012 at 8:58 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> After a quick look at cmake, it seems reasonable. Please use feature tests
> rather than OS tests. (By that I mean that if Solaris needs "foobar.h" and
> RHEL needs "sys/foobar.h" to get the definition for foobar, the code should
> ifdef whether they need sys/foobar.h and not if it is solaris or not.)
>
> On a related note, on what platforms does the native code currently compile
> and work?
>
> I've compiled and tested on:
>  RHEL & CentOS 5
>  RHEL & CentOS 6
>
> I've heard of it compiling on:
>  Solaris
>
> What other ones are out there?

Bigtop routinely compiles and tests native code on the following list
of platforms:
    http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-Hadoop/

Thanks,
Roman.

On Wed, May 23, 2012 at 8:58 AM, Owen O'Malley <[EMAIL PROTECTED]> wrote:
> After a quick look at cmake, it seems reasonable. Please use feature tests
> rather than OS tests. (By that I mean that if Solaris needs "foobar.h" and
> RHEL needs "sys/foobar.h" to get the definition for foobar, the code should
> ifdef whether they need sys/foobar.h and not if it is solaris or not.)
>
> On a related note, on what platforms does the native code currently compile
> and work?
>
> I've compiled and tested on:
>  RHEL & CentOS 5
>  RHEL & CentOS 6
>
> I've heard of it compiling on:
>  Solaris
>
> What other ones are out there?

Also multiple versions of OUL, SLES, Debian, and Ubuntu.

I am +1 with this change as well. CMake is a good choice. My other
favorite project, KDE, uses CMake as well and it has worked great
there ever since 2007.

On Tue, May 22, 2012 at 8:40 AM, Eli Collins <[EMAIL PROTECTED]> wrote:
> +1   having a build tool that supports multiple platforms is worth the
> dependency. I've also had good experiences with cmake.
>
>
> On Mon, May 21, 2012 at 6:00 PM, Colin McCabe <[EMAIL PROTECTED]> wrote:
>> Hi all,
>>
>> We'd like to use CMake instead of autotools to build native (C/C++) code in
>> Hadoop.  There are a lot of reasons to want to do this.  For one thing, it is
>> not feasible to use autotools on the Windows platform, because it depends on
>> UNIX shell scripts, the m4 macro processor, and some other pieces of
>> infrastructure which are not present on Windows.
>>
>> For another thing, CMake builds are substantially simpler and faster, because
>> there is only one layer of generated code.  With autotools, you have automake
>> generating m4 code which autoconf reads, which it uses to generate a UNIX shell
>> script, which then generates another UNIX shell script, which eventually
>> generates Makefiles.  CMake simply generates Makefiles out of CMakeLists.txt
>> files-- much simpler to understand and debug, and much faster.
>> CMake is a lot easier to learn.
>>
>> automake error messages can be very, very confusing.  This is because you are
>> essentially debugging a pile of shell scripts and macros, rather than a
>> coherent whole.  So you see error messages like "autoreconf: cannot empty
>> /tmp/ar0.4849: Is a directory" or "Can't locate object method "path" via
>> package "Autom4te... and so forth.  CMake error messages come from the CMake
>> application and they almost always immediately point you to the problem.
>>
>> From a build point of view, the net result of adopting CMake would be that you
>> would no longer need automake and related programs installed to build the
>> native parts of Hadoop.  Instead, you would need CMake installed.  CMake is
>> packaged by Red Hat, even in RHEL5, so it shouldn't be difficult to install
>> locally.  It's also available for Mac OS X and Windows, as I mentioned earlier.
>>
>> The JIRA for this work is at https://issues.apache.org/jira/browse/HADOOP-8368
>> Thanks for reading.
>>
>> sincerely,
>> Colin

--
Harsh J

+1, especially for support of Win native builds.

On Tue, May 22, 2012 at 5:07 AM, Harsh J <[EMAIL PROTECTED]> wrote:

> I am +1 with this change as well. CMake is a good choice. My other
> favorite project, KDE, uses CMake as well and it has worked great
> there ever since 2007.
>
> On Tue, May 22, 2012 at 8:40 AM, Eli Collins <[EMAIL PROTECTED]> wrote:
> > +1   having a build tool that supports multiple platforms is worth the
> > dependency. I've also had good experiences with cmake.
> >
> >
> > On Mon, May 21, 2012 at 6:00 PM, Colin McCabe <[EMAIL PROTECTED]>
> wrote:
> >> Hi all,
> >>
> >> We'd like to use CMake instead of autotools to build native (C/C++)
> code in
> >> Hadoop.  There are a lot of reasons to want to do this.  For one thing,
> it is
> >> not feasible to use autotools on the Windows platform, because it
> depends on
> >> UNIX shell scripts, the m4 macro processor, and some other pieces of
> >> infrastructure which are not present on Windows.
> >>
> >> For another thing, CMake builds are substantially simpler and faster,
> because
> >> there is only one layer of generated code.  With autotools, you have
> automake
> >> generating m4 code which autoconf reads, which it uses to generate a
> UNIX shell
> >> script, which then generates another UNIX shell script, which eventually
> >> generates Makefiles.  CMake simply generates Makefiles out of
> CMakeLists.txt
> >> files-- much simpler to understand and debug, and much faster.
> >> CMake is a lot easier to learn.
> >>
> >> automake error messages can be very, very confusing.  This is because
> you are
> >> essentially debugging a pile of shell scripts and macros, rather than a
> >> coherent whole.  So you see error messages like "autoreconf: cannot
> empty
> >> /tmp/ar0.4849: Is a directory" or "Can't locate object method "path" via
> >> package "Autom4te... and so forth.  CMake error messages come from the
> CMake
> >> application and they almost always immediately point you to the problem.
> >>
> >> From a build point of view, the net result of adopting CMake would be
> that you
> >> would no longer need automake and related programs installed to build
> the
> >> native parts of Hadoop.  Instead, you would need CMake installed.
>  CMake is
> >> packaged by Red Hat, even in RHEL5, so it shouldn't be difficult to
> install
> >> locally.  It's also available for Mac OS X and Windows, as I mentioned
> earlier.
> >>
> >> The JIRA for this work is at
> https://issues.apache.org/jira/browse/HADOOP-8368
> >> Thanks for reading.
> >>
> >> sincerely,
> >> Colin
>
>
>
> --
> Harsh J
>