Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Setting up Snappy compression in Hadoop


Copy link to this message
-
RE: Setting up Snappy compression in Hadoop
java8964 2014-01-02, 16:02
If you really confirmed that libsnappy.so.1 is in the correct location, and being loaded into java library path, working in your test program, but still didn't work in MR, there is one another possibility which was puzzling me before.
How do you get the libhadoop.so in your hadoop environment? Did you compile it by yourself, or is it  from some vendors?
You want to make sure the Java native methods of invoking snappy is also being compiled and available in libhadoop.so.
For example, the following command will prove it:
$ nm ./libhadoop.so | grep snappy00000000000035c0 T Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_compressBytesDirect0000000000003960 T Java_org_apache_hadoop_io_compress_snappy_SnappyCompressor_initIDs0000000000003bb0 T Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_decompressBytesDirect0000000000003f60 T Java_org_apache_hadoop_io_compress_snappy_SnappyDecompressor_initIDs0000000000206cf0 b dlsym_snappy_compress0000000000206d20 b dlsym_snappy_uncompress
Without these Java native methods being compiled and available in the libhadoop.so, MR runtime will also complain that "native snappy library not available".
Yong
Date: Thu, 2 Jan 2014 13:37:46 +0200
Subject: Re: Setting up Snappy compression in Hadoop
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]

I did everything mentioned in the link Ted mentioned, and the test actually works, but using Snappy for MapReduce map output compression still fails with "native snappy library not available".
On Wed, Jan 1, 2014 at 6:37 PM, bharath vissapragada <[EMAIL PROTECTED]> wrote:

Did you build it for your platform? You can do an "ldd" on the .so file to check if the dependent libs are present. Also make sure you placed it in the right directory for your platform (Linux-amd64-64 or Linux-i386-32)
On Wed, Jan 1, 2014 at 10:02 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

Please take a look at http://hbase.apache.org/book.html#snappy.compression
Cheers
On Wed, Jan 1, 2014 at 8:05 AM, Amit Sela <[EMAIL PROTECTED]> wrote:
Hi all,
I'm running on Hadoop 1.0.4 and I'd like to use Snappy for map output compression.I'm adding the configurations:
configuration.setBoolean("mapred.compress.map.output", true);

configuration.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");

And I've added libsnappy.so.1 to $HADOOP_HOME/lib/native/Linux-amd64-64/

Still, all map tasks fail with "native snappy library not available".
Could anyone elaborate on how to install Snappy for Hadoop ?

Thanks,
Amit.