Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> RE: how to turn on NativeIO.posixFadviseIfPossible


Copy link to this message
-
RE: how to turn on NativeIO.posixFadviseIfPossible
This looks like a compilation problem on the native hadoop libraries.

Please locate the libhadoop.so library on your system and run
[shell]  objdump -Tt libhadoop.so | grep -I fadvise

If you don't see something like the following *the GLIBC* part  (that means the system where the share lib was compiled did not have it)

00000000000056a0 g     F .text  00000000000000a3              Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
0000000000000000       F *UND*  0000000000000000              posix_fadvise@@GLIBC_2.2.5
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
00000000000056a0 g    DF .text  00000000000000a3  Base        Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise

Note: objdump is from binutils- rpm  (you can use yum install to install it if you don't have it)
-----Original Message-----
From: Jun Li [mailto:[EMAIL PROTECTED]]
Sent: Friday, June 21, 2013 1:35 AM
To: [EMAIL PROTECTED]
Subject: how to turn on NativeIO.posixFadviseIfPossible

Hi,

I downloaded the current stable version from the Apache Hadoop web site, hadoop-1.1.2. My machine is an AMD-based machine and Redhat Enterprise 6.1. The detailed Linux kernel version is:
2.6.32-131.0.15.el6.x86_64

 I ran the TestNativeIO.java under the distribution directory of "test/org/apache/hadoop/io/nativeio/TestNativeIO.java" and tried to understand how NativeIO.posixFadviseIfPossible behaves, in particular, to check whether "posix_fadvise" is turned on or not. I am interested in this call as it is used in Read-Ahead-Pool to cache data to the OS's buffer cache.  The following is the test case that I ran:

@Test
  public void testPosixFadvise() throws Exception {
    FileInputStream fis = new FileInputStream("/dev/zero");
    try {
      NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0,
                             NativeIO.POSIX_FADV_SEQUENTIAL);
    } catch (NativeIOException noe) {
      // we should just skip the unit test on machines where we don't
      // have fadvise support
      assumeTrue(false);
    } finally {
      fis.close();
    }

However, when I stepped into the code and reached "NativeIO.java"
under the package of "org.apache.hadoop.io.nativeio",  in the particular call below:

public static void posixFadviseIfPossible(
      FileDescriptor fd, long offset, long len, int flags)
      throws NativeIOException {

    if (nativeLoaded && fadvisePossible) {
      try {
        posix_fadvise(fd, offset, len, flags);
      } catch (UnsupportedOperationException uoe) {
        fadvisePossible = false;
      } catch (UnsatisfiedLinkError ule) {
        fadvisePossible = false;
      }
    }
  }

The call to "posix_fadvise"  threw the "UnsupportedOperationException"
exception.

I further traced to the native library, and in the code "NativeIO.c", I found

JNIEXPORT void JNICALL
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise(
  JNIEnv *env, jclass clazz,
  jobject fd_object, jlong offset, jlong len, jint flags) { #ifndef HAVE_POSIX_FADVISE
  THROW(env, "java/lang/UnsupportedOperationException",
        "fadvise support not available"); #else

...
}

I believe that the problem of throwing the exception is because "HAVE_POSIX_FADVISE" is not defined.  I made sure that the native IO library is loaded properly in the Java code, as I can successfully run the other test cases in "TestNativeIO.java".

So my question is: should I re-compile the "libhadoop" in order to get the version of the shared library that can have "HAVE_POSIX_FADVISE"
turned on? Or by default, FADVISE is turned on already?

Thank you!
+
Leo Leung 2013-06-21, 19:35
+
Leo Leung 2013-06-22, 23:42