Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: how to turn on NativeIO.posixFadviseIfPossible


Copy link to this message
-
Re: how to turn on NativeIO.posixFadviseIfPossible
Jun Li 2013-06-22, 10:25
Hi Leo,

Thanks for the detailed instruction. I started from the clean downloaded
hadoop1.1.12 copy and then I issue the command: ant -Dcompile.native=true.

And with the configure.ac that comes with the downloaded package, without
any changes, the following is the compilation output:
     [exec] checking fcntl.h usability... yes
     [exec] checking fcntl.h presence... yes
     [exec] checking for fcntl.h... yes
     [exec] checking for stdlib.h... (cached) yes
     [exec] checking for string.h... (cached) yes
     [exec] checking for unistd.h... (cached) yes
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for posix_fadvise... no
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for sync_file_range... no
     [exec] checking for an ANSI C-conforming const... yes
     [exec] checking for memset... no
     [exec] checking whether strerror_r is declared... yes
     [exec] checking for strerror_r... no

I have difficulty to follow the "+" and "-" relate patching sequence that
you stated in your email earlier. I have attached the "configure.ac" file
that I copied out of the hadoop1.1.12/src/native directory. Could you help
me to construct the configure.ac with the correct patching that you
described, using the one that I attached in this email, and send back to
me? Then I will use the patched one to build the native library.

Regards,

Jun

On Fri, Jun 21, 2013 at 12:35 PM, Leo Leung <[EMAIL PROTECTED]> wrote:

>  Hi Jun:****
>
> ** **
>
> Looks like it is not in the apache hadoop 1.x release binaries,  this
> probably will have to be answered by the apache hadoop 1.x release team***
> *
>
> ** **
>
> To build the native libs:****
>
> ** **
>
> **1)      **Get the source code and install the dependencies to compile
> hadoop****
>
> **2)      **You’ll have to setup the env for it,  such as JAVA_HOME. Gcc
> installed etc.****
>
> **3)      **Got to <hadoop-source dir>     ant [veryclean]
> compile-core-native  >compile.out****
>
> ** **
>
> Verify with ****
>
> [branch-1]$ objdump -Tt ./build/native/Linux-amd64-64/lib/libhadoop.so |
> grep fadv****
>
> 00000000000056a0 g     F .text  00000000000000a3
> Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise****
>
> 0000000000000000       F *UND*  0000000000000000
> posix_fadvise@@GLIBC_2.2.5****
>
> 0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
> ****
>
> 00000000000056a0 g    DF .text  00000000000000a3  Base
> Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
>
>
> ****
>
> If you get an error about  “make distclean” error****
>
> Just cd in src/native and run  make distclean (to cleanup )  ****
>
> ** **
>
> Example of ant output that shows checking for fadvise ****
>
>      [exec] checking for fcntl.h... yes****
>
>      [exec] checking for stdlib.h... (cached) yes****
>
>      [exec] checking for string.h... (cached) yes****
>
>      [exec] checking for unistd.h... (cached) yes****
>
>      [exec] checking for fcntl.h... (cached) yes****
>
>      [exec] checking for posix_fadvise... yes****
>
>      [exec] checking for fcntl.h... (cached) yes****
>
>      [exec] checking for sync_file_range... yes****
>
> ** **
>
> If you see “no” for posix_fadvise or “sync_file_range” ****
>
> And you are sure you have posix_fdavise support (RHEL 6.x or CentOS 6.x)**
> **
>
> ** **
>
> you may need to move the section on src/native/configure.ac****
>
> Try to modify the configure.ac
> --- src/native/configure.ac     (revision 1495498)****
>
> +++ src/native/configure.ac     (working copy)****
>
> @@ -47,6 +47,21 @@****
>
> AC_PROG_CC****
>
> AC_PROG_LIBTOOL****
>
> ** **
>
> +dnl check for posix_fadvise****
>
> +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)])****
>
> +****
>
> +dnl check for sync_file_range****
>
> +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)])****
>
> +****
>
> +# Checks for typedefs, structures, and compiler characteristics.****
>
> +AC_C_CONST****
>
> +****
>
> +# Checks for library functions.****