Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Issue with loading the Snappy Codec


Copy link to this message
-
Re: Issue with loading the Snappy Codec

Hadoop has integrated snappy via installed native libraries instead of snappy-java.jar (ref https://issues.apache.org/jira/browse/HADOOP-7206)
 - You need to have the snappy system libraries (snappy and snappy-devel) installed before you compile hadoop. (RPMs are available on the web, http://pkgs.org/centos-5-rhel-5/epel-i386/21/ for example)
 - When you build hadoop, you will need to compile the native libraries(by passing -Dcompile.native=true to ant) to avail snappy support.
 - You also need to make sure that snappy system library is available on the library path for all mapreduce tasks at runtime. Usually if you install them on /usr/lib or /usr/local/lib, it should work.

HTH,
+Vinod

On Apr 14, 2012, at 4:36 AM, Bas Hickendorff wrote:

> Hello,
>
> When I start a map-reduce job, it starts, and after a short while,
> fails with the error below (SnappyCodec not found).
>
> I am currently starting the job from other Java code (so the Hadoop
> executable in the bin directory is not used anymore), but in principle
> this seems to work (in the admin of the Jobtracker the job shows up
> when it starts). However after a short while the map task fails with:
>
>
> java.lang.IllegalArgumentException: Compression codec
> org.apache.hadoop.io.compress.SnappyCodec not found.
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
> at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:62)
> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:522)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.io.compress.SnappyCodec
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:334)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89)
> ... 10 more
>
>
> I confirmed that the SnappyCodec class is present in the
> hadoop-core-1.0.2.jar, and the snappy-java-1.0.4.1.jar is present as
> well. The directory of those jars is on the HADOOP_CLASSPATH, but it
> seems it still cannot find it. I also checked that the config files of
> Hadoop are read. I run all nodes on localhost.
>
> Any suggestions on what could be the cause of the issue?
>
> Regards,
>
> Bas
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB