Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - how to turn on NativeIO.posixFadviseIfPossible


+
Jun Li 2013-06-21, 08:35
Copy link to this message
-
Re: how to turn on NativeIO.posixFadviseIfPossible
Jun Li 2013-06-21, 16:43
Hi Leo,

Following your instruction,  the following is what I got:

[junli@mercoop-26 Linux-amd64-64]$ objdump -Tt libhadoop.so | grep -I
fadvise
0000000000004d70 g     F .text    000000000000006f
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
0000000000004d70 g    DF .text    000000000000006f  Base
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise

Apparently the part of "the GLIBC" is not there

The libhadoop.so is the one that I downloaded as part of the  tar.gz of
hadoop.1.1.2 from the Hadoop Apache web site.

Could you let me know how I can use the source code directory in the
downloaded Hadoop Apache package, to re-compile the libhadoop.so and to
make sure that the fadvise call is able to get referenced?

Regards,

Jun

On Fri, Jun 21, 2013 at 9:19 AM, Leo Leung <[EMAIL PROTECTED]> wrote:

> This looks like a compilation problem on the native hadoop libraries.
>
> Please locate the libhadoop.so library on your system and run
> [shell]  objdump -Tt libhadoop.so | grep -I fadvise
>
> If you don't see something like the following *the GLIBC* part  (that
> means the system where the share lib was compiled did not have it)
>
> 00000000000056a0 g     F .text  00000000000000a3
>  Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
> 0000000000000000       F *UND*  0000000000000000
>  posix_fadvise@@GLIBC_2.2.5
> 0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
> 00000000000056a0 g    DF .text  00000000000000a3  Base
>  Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
>
> Note: objdump is from binutils- rpm  (you can use yum install to install
> it if you don't have it)
>
>
> -----Original Message-----
> From: Jun Li [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 21, 2013 1:35 AM
> To: [EMAIL PROTECTED]
> Subject: how to turn on NativeIO.posixFadviseIfPossible
>
> Hi,
>
> I downloaded the current stable version from the Apache Hadoop web site,
> hadoop-1.1.2. My machine is an AMD-based machine and Redhat Enterprise 6.1.
> The detailed Linux kernel version is:
> 2.6.32-131.0.15.el6.x86_64
>
>  I ran the TestNativeIO.java under the distribution directory of
> "test/org/apache/hadoop/io/nativeio/TestNativeIO.java" and tried to
> understand how NativeIO.posixFadviseIfPossible behaves, in particular, to
> check whether "posix_fadvise" is turned on or not. I am interested in this
> call as it is used in Read-Ahead-Pool to cache data to the OS's buffer
> cache.  The following is the test case that I ran:
>
> @Test
>   public void testPosixFadvise() throws Exception {
>     FileInputStream fis = new FileInputStream("/dev/zero");
>     try {
>       NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0,
>                              NativeIO.POSIX_FADV_SEQUENTIAL);
>     } catch (NativeIOException noe) {
>       // we should just skip the unit test on machines where we don't
>       // have fadvise support
>       assumeTrue(false);
>     } finally {
>       fis.close();
>     }
>
> However, when I stepped into the code and reached "NativeIO.java"
> under the package of "org.apache.hadoop.io.nativeio",  in the particular
> call below:
>
> public static void posixFadviseIfPossible(
>       FileDescriptor fd, long offset, long len, int flags)
>       throws NativeIOException {
>
>     if (nativeLoaded && fadvisePossible) {
>       try {
>         posix_fadvise(fd, offset, len, flags);
>       } catch (UnsupportedOperationException uoe) {
>         fadvisePossible = false;
>       } catch (UnsatisfiedLinkError ule) {
>         fadvisePossible = false;
>       }
>     }
>   }
>
> The call to "posix_fadvise"  threw the "UnsupportedOperationException"
> exception.
>
> I further traced to the native library, and in the code "NativeIO.c", I
> found
>
> JNIEXPORT void JNICALL
> Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise(
>   JNIEnv *env, jclass clazz,
>   jobject fd_object, jlong offset, jlong len, jint flags) { #ifndef
> HAVE_POSIX_FADVISE
>   THROW(env, "java/lang/UnsupportedOperationException",
+
Jun Li 2013-06-24, 06:23