Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - RE: how to turn on NativeIO.posixFadviseIfPossible


+
Leo Leung 2013-06-21, 16:19
+
Leo Leung 2013-06-21, 19:35
Copy link to this message
-
RE: how to turn on NativeIO.posixFadviseIfPossible
Leo Leung 2013-06-22, 23:42
Hi Jun,

  Try moving the section between:
dnl Check for headers needed by the native Group....
  [...include everything in between inclusive ...]
AC_FUNC_STRERROR_R

to higher place (insert before)

# Checks for libraries.
dnl Check for '-ldl'

Hope this helps :)
From: Jun Li [mailto:[EMAIL PROTECTED]]
Sent: Saturday, June 22, 2013 3:26 AM
To: [EMAIL PROTECTED]
Subject: Re: how to turn on NativeIO.posixFadviseIfPossible

Hi Leo,

Thanks for the detailed instruction. I started from the clean downloaded hadoop1.1.12 copy and then I issue the command: ant -Dcompile.native=true.

And with the configure.ac<http://configure.ac> that comes with the downloaded package, without any changes, the following is the compilation output:
     [exec] checking fcntl.h usability... yes
     [exec] checking fcntl.h presence... yes
     [exec] checking for fcntl.h... yes
     [exec] checking for stdlib.h... (cached) yes
     [exec] checking for string.h... (cached) yes
     [exec] checking for unistd.h... (cached) yes
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for posix_fadvise... no
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for sync_file_range... no
     [exec] checking for an ANSI C-conforming const... yes
     [exec] checking for memset... no
     [exec] checking whether strerror_r is declared... yes
     [exec] checking for strerror_r... no
I have difficulty to follow the "+" and "-" relate patching sequence that you stated in your email earlier. I have attached the "configure.ac<http://configure.ac>" file that I copied out of the hadoop1.1.12/src/native directory. Could you help me to construct the configure.ac<http://configure.ac> with the correct patching that you described, using the one that I attached in this email, and send back to me? Then I will use the patched one to build the native library.
Regards,

Jun

On Fri, Jun 21, 2013 at 12:35 PM, Leo Leung <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Hi Jun:

Looks like it is not in the apache hadoop 1.x release binaries,  this probably will have to be answered by the apache hadoop 1.x release team

To build the native libs:
1)      Get the source code and install the dependencies to compile hadoop

2)      You'll have to setup the env for it,  such as JAVA_HOME. Gcc installed etc.

3)      Got to <hadoop-source dir>     ant [veryclean] compile-core-native  >compile.out

Verify with
[branch-1]$ objdump -Tt ./build/native/Linux-amd64-64/lib/libhadoop.so | grep fadv
00000000000056a0 g     F .text  00000000000000a3              Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
0000000000000000       F *UND*  0000000000000000              posix_fadvise@@GLIBC_2.2.5
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
00000000000056a0 g    DF .text  00000000000000a3  Base        Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise

If you get an error about  "make distclean" error
Just cd in src/native and run  make distclean (to cleanup )

Example of ant output that shows checking for fadvise
     [exec] checking for fcntl.h... yes
     [exec] checking for stdlib.h... (cached) yes
     [exec] checking for string.h... (cached) yes
     [exec] checking for unistd.h... (cached) yes
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for posix_fadvise... yes
     [exec] checking for fcntl.h... (cached) yes
     [exec] checking for sync_file_range... yes

If you see "no" for posix_fadvise or "sync_file_range"
And you are sure you have posix_fdavise support (RHEL 6.x or CentOS 6.x)

you may need to move the section on src/native/configure.ac<http://configure.ac>
Try to modify the configure.ac<http://configure.ac>
--- src/native/configure.ac<http://configure.ac>     (revision 1495498)
+++ src/native/configure.ac<http://configure.ac>     (working copy)
@@ -47,6 +47,21 @@
AC_PROG_CC
AC_PROG_LIBTOOL

+dnl check for posix_fadvise
+AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)])
+
+dnl check for sync_file_range
+AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)])
+
+# Checks for typedefs, structures, and compiler characteristics.
+AC_C_CONST
+
+# Checks for library functions.
+AC_CHECK_FUNCS([memset])
+
+# Check for nonstandard STRERROR_R
+AC_FUNC_STRERROR_R
+
# Checks for libraries.
dnl Check for '-ldl'
AC_CHECK_LIB([dl], [dlopen])
@@ -104,21 +119,6 @@
dnl Check for headers needed by the native Group resolution implementation
AC_CHECK_HEADERS([fcntl.h stdlib.h string.h unistd.h], [], AC_MSG_ERROR(Some system headers not found... please ensure their presence on your platform.))

-dnl check for posix_fadvise
-AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(posix_fadvise)])
-
-dnl check for sync_file_range
-AC_CHECK_HEADERS(fcntl.h, [AC_CHECK_FUNCS(sync_file_range)])
-
-# Checks for typedefs, structures, and compiler characteristics.
-AC_C_CONST
-
-# Checks for library functions.
-AC_CHECK_FUNCS([memset])
-
-# Check for nonstandard STRERROR_R
-AC_FUNC_STRERROR_R
-
AC_CONFIG_FILES([Makefile])
AC_OUTPUT
Hope this helps
From: Jun Li [mailto:[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>]
Sent: Friday, June 21, 2013 9:44 AM
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Subject: Re: how to turn on NativeIO.posixFadviseIfPossible

Hi Leo,

Following your instruction,  the following is what I got:

[junli@mercoop-26 Linux-amd64-64]$ objdump -Tt libhadoop.so | grep -I fadvise
0000000000004d70 g     F .text    000000000000006f              Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
0000000000004d70 g    DF .text    000000000000006f  Base        Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
Apparently the part of "the GLIBC" is not there
The libhadoop.so is the one that I downloaded as part of the  tar.gz of hadoop.1.1.2 from the Hadoop Apache web site.
Could you let me know how I can use the source code directory in the downloaded Hadoop Apache package, to re-compile the libhadoop.so and