Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: how to turn on NativeIO.posixFadviseIfPossible


Copy link to this message
-
Re: how to turn on NativeIO.posixFadviseIfPossible
Hi Leo,

Thanks for the detailed instruction. I started from the clean downloaded
hadoop1.1.12 copy and then I issue the command: ant -Dcompile.native=true.

And with the configure.ac that comes with the downloaded package, without
any changes, the following is the compilation output:
     [exec] checking fcntl.h usability... yes
     [exec] checking fcntl.h presence... yes
     [exec] checking for fcntl.h... yes
     [exec] checking for stdlib.h... (cached) yes
     [exec] checking for string.h... (cached) yes
     [exec] checking for unistd.h... (cached) yes
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for posix_fadvise... no
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for sync_file_range... no
     [exec] checking for an ANSI C-conforming const... yes
     [exec] checking for memset... no
     [exec] checking whether strerror_r is declared... yes
     [exec] checking for strerror_r... no

I have difficulty to follow the "+" and "-" relate patching sequence that
you stated in your email earlier. I have attached the "configure.ac" file
that I copied out of the hadoop1.1.12/src/native directory. Could you help
me to construct the configure.ac with the correct patching that you
described, using the one that I attached in this email, and send back to
me? Then I will use the patched one to build the native library.

Regards,

Jun

On Fri, Jun 21, 2013 at 12:35 PM, Leo Leung <[EMAIL PROTECTED]> wrote:

>  Hi Jun:****
>
> ** **
>
> Looks like it is not in the apache hadoop 1.x release binaries,  this
> probably will have to be answered by the apache hadoop 1.x release team***
> *
>
> ** **
>
> To build the native libs:****
>
> ** **
>
> **1)      **Get the source code and install the dependencies to compile
> hadoop****
>
> **2)      **You’ll have to setup the env for it,  such as JAVA_HOME. Gcc
> installed etc.****
>
> **3)      **Got to <hadoop-source dir>     ant [veryclean]
> compile-core-native  >compile.out****
>
> ** **
>
> Verify with ****
>
> [branch-1]$ objdump -Tt ./build/native/Linux-amd64-64/lib/libhadoop.so |
> grep fadv****
>
> 00000000000056a0 g     F .text  00000000000000a3
> Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise****
>
> 0000000000000000       F *UND*  0000000000000000
> posix_fadvise@@GLIBC_2.2.5****
>
> 0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
> ****
>
> 00000000000056a0 g    DF .text  00000000000000a3  Base
> Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
>
>
> ****
>
> If you get an error about  “make distclean” error****
>
> Just cd in src/native and run  make distclean (to cleanup )  ****
>
> ** **
>
> Example of ant output that shows checking for fadvise ****
>
>      [exec] checking for fcntl.h... yes****
>
>      [exec] checking for stdlib.h... (cached) yes****
>
>      [exec] checking for string.h... (cached) yes****
>
>      [exec] checking for unistd.h... (cached) yes****
>
>      [exec] checking for fcntl.h... (cached) yes****
>
>      [exec] checking for posix_fadvise... yes****
>
>      [exec] checking for fcntl.h... (cached) yes****
>
>      [exec] checking for sync_file_range... yes****
>
> ** **
>
> If you see “no” for posix_fadvise or “sync_file_range” ****
>
> And you are sure you have posix_fdavise support (RHEL 6.x or CentOS 6.x)**
> **
>
> ** **
>
> you may need to move the section on src/native/configure.ac****
>
> Try to modify the configure.ac
> --- src/native/configure.ac     (revision 1495498)****
>
> +++ src/native/configure.ac     (working copy)****
>
> @@ -47,6 +47,21 @@****
>
> AC_PROG_CC****
>
> AC_PROG_LIBTOOL****
>
> ** **
>
> +dnl check for posix_fadvise****
>
> +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)])****
>
> +****
>
> +dnl check for sync_file_range****
>
> +AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)])****
>
> +****
>
> +# Checks for typedefs, structures, and compiler characteristics.****
>
> +AC_C_CONST****
>
> +****
>
> +# Checks for library functions.****
+
Leo Leung 2013-06-24, 17:22
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB