Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> how to turn on NativeIO.posixFadviseIfPossible


+
Jun Li 2013-06-21, 08:35
Copy link to this message
-
Re: how to turn on NativeIO.posixFadviseIfPossible
Hi Leo,

Following your instruction,  the following is what I got:

[junli@mercoop-26 Linux-amd64-64]$ objdump -Tt libhadoop.so | grep -I
fadvise
0000000000004d70 g     F .text    000000000000006f
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
0000000000004d70 g    DF .text    000000000000006f  Base
Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise

Apparently the part of "the GLIBC" is not there

The libhadoop.so is the one that I downloaded as part of the  tar.gz of
hadoop.1.1.2 from the Hadoop Apache web site.

Could you let me know how I can use the source code directory in the
downloaded Hadoop Apache package, to re-compile the libhadoop.so and to
make sure that the fadvise call is able to get referenced?

Regards,

Jun

On Fri, Jun 21, 2013 at 9:19 AM, Leo Leung <[EMAIL PROTECTED]> wrote:

> This looks like a compilation problem on the native hadoop libraries.
>
> Please locate the libhadoop.so library on your system and run
> [shell]  objdump -Tt libhadoop.so | grep -I fadvise
>
> If you don't see something like the following *the GLIBC* part  (that
> means the system where the share lib was compiled did not have it)
>
> 00000000000056a0 g     F .text  00000000000000a3
>  Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
> 0000000000000000       F *UND*  0000000000000000
>  posix_fadvise@@GLIBC_2.2.5
> 0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 posix_fadvise
> 00000000000056a0 g    DF .text  00000000000000a3  Base
>  Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise
>
> Note: objdump is from binutils- rpm  (you can use yum install to install
> it if you don't have it)
>
>
> -----Original Message-----
> From: Jun Li [mailto:[EMAIL PROTECTED]]
> Sent: Friday, June 21, 2013 1:35 AM
> To: [EMAIL PROTECTED]
> Subject: how to turn on NativeIO.posixFadviseIfPossible
>
> Hi,
>
> I downloaded the current stable version from the Apache Hadoop web site,
> hadoop-1.1.2. My machine is an AMD-based machine and Redhat Enterprise 6.1.
> The detailed Linux kernel version is:
> 2.6.32-131.0.15.el6.x86_64
>
>  I ran the TestNativeIO.java under the distribution directory of
> "test/org/apache/hadoop/io/nativeio/TestNativeIO.java" and tried to
> understand how NativeIO.posixFadviseIfPossible behaves, in particular, to
> check whether "posix_fadvise" is turned on or not. I am interested in this
> call as it is used in Read-Ahead-Pool to cache data to the OS's buffer
> cache.  The following is the test case that I ran:
>
> @Test
>   public void testPosixFadvise() throws Exception {
>     FileInputStream fis = new FileInputStream("/dev/zero");
>     try {
>       NativeIO.posixFadviseIfPossible(fis.getFD(), 0, 0,
>                              NativeIO.POSIX_FADV_SEQUENTIAL);
>     } catch (NativeIOException noe) {
>       // we should just skip the unit test on machines where we don't
>       // have fadvise support
>       assumeTrue(false);
>     } finally {
>       fis.close();
>     }
>
> However, when I stepped into the code and reached "NativeIO.java"
> under the package of "org.apache.hadoop.io.nativeio",  in the particular
> call below:
>
> public static void posixFadviseIfPossible(
>       FileDescriptor fd, long offset, long len, int flags)
>       throws NativeIOException {
>
>     if (nativeLoaded && fadvisePossible) {
>       try {
>         posix_fadvise(fd, offset, len, flags);
>       } catch (UnsupportedOperationException uoe) {
>         fadvisePossible = false;
>       } catch (UnsatisfiedLinkError ule) {
>         fadvisePossible = false;
>       }
>     }
>   }
>
> The call to "posix_fadvise"  threw the "UnsupportedOperationException"
> exception.
>
> I further traced to the native library, and in the code "NativeIO.c", I
> found
>
> JNIEXPORT void JNICALL
> Java_org_apache_hadoop_io_nativeio_NativeIO_posix_1fadvise(
>   JNIEnv *env, jclass clazz,
>   jobject fd_object, jlong offset, jlong len, jint flags) { #ifndef
> HAVE_POSIX_FADVISE
>   THROW(env, "java/lang/UnsupportedOperationException",
+
Jun Li 2013-06-24, 06:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB