Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> enable snappy on hadoop 1.1.1


Copy link to this message
-
enable snappy on hadoop 1.1.1
Hi,
I am using hadoop 1.1.1. I want to test to see the snappy compression with hadoop, but I have some problems to make it work on my Linux environment.
I am using opensuse 12.3 x86_64.
First, when I tried to enable snappy in hadoop 1.1.1 by:
            conf.setBoolean("mapred.compress.map.output", true);            conf.set("mapred.output.compression.type", "RECORD");            conf.set("mapred.map.output.compression.codec", "org.apache.hadoop.io.compress.SnappyCodec");
I got the following error in my test MR job:
Exception in thread "main" java.lang.RuntimeException: native snappy library not available
So I download the snappy 1.1.0 from https://code.google.com/p/snappy/, compile it and install it successfully under /opt/snappy-1.1.0, and then I link the /opt/snappy-1.1.0/lib64/libsnappy.so to /user/lib64/libsnappy.so
Now after I restart the hadoop and tried my test MR job again, this time, it didn't give me the originally error, but a new error like this:
Error: java.lang.UnsatisfiedLinkError: org/apache/hadoop/io/compress/snappy/SnappyCompressor.compressBytesDirect()I at org.apache.hadoop.io.compress.snappy.SnappyCompressor.compress(SnappyCompressor.java:229) at org.apache.hadoop.io.compress.BlockCompressorStream.compress(BlockCompressorStream.java:141) at org.apache.hadoop.io.compress.BlockCompressorStream.finish(BlockCompressorStream.java:135) at org.apache.hadoop.mapred.IFile$Writer.close(IFile.java:135) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1450) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:852) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1343)
I wrote a test problem, like hadoop did load the library:
System.loadlibrary("snappy")
it works fine in my test program.
I don't know why at runtime, the Class SnappyCompressor.compressByteDirect() gave back that kind of error. From the source code, it looks like a native c program from here
https://code.google.com/p/hadoop-snappy/source/browse/trunk/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
Any idea why this is happening on my environment?
Thanks
Yong