Hi,

I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy is included in their hadoop-0.20-native package and it is also installed on each of the nodes.

But when I run a mapreduce task with "mapred.map.output.compression.codec" set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:

java.lang.RuntimeException: native snappy library not available
Any idea why?

Thanks,
Marek M.

I believe that the mapred.child.env only concerns the spawned child
process where the map is run.  You probably need to put this path into the
system property where the map task demon runs and possibly where the job
controller runs.

Zhu, Guojun
Modeling Sr Graduate
571-3824370
[EMAIL PROTECTED]
Financial Engineering
Freddie Mac

   Marek Miglinski <[EMAIL PROTECTED]>
   06/11/2012 09:54 AM
   Please respond to
[EMAIL PROTECTED]
To
"[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
cc

Subject
snappy codec
Hi,

I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set
to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with
libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy
is included in their hadoop-0.20-native package and it is also installed
on each of the nodes.

But when I run a mapreduce task with "mapred.map.output.compression.codec"
set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:

java.lang.RuntimeException: native snappy library not available
Any idea why?

Thanks,
Marek M.

use java lib path

On Mon, Jun 11, 2012 at 7:12 AM, GUOJUN Zhu <[EMAIL PROTECTED]>wrote:

>
> I believe that the mapred.child.env only concerns the spawned child
> process where the map is run.  You probably need to put this path into the
> system property where the map task demon runs and possibly where the job
> controller runs.
>
> Zhu, Guojun
> Modeling Sr Graduate
> 571-3824370
> [EMAIL PROTECTED]
> Financial Engineering
> Freddie Mac
>
>
>      *Marek Miglinski <[EMAIL PROTECTED]>*
>
>    06/11/2012 09:54 AM
>      Please respond to
> [EMAIL PROTECTED]
>
>    To
> "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> cc
>   Subject
> snappy codec
>
>
>
>
> Hi,
>
> I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set
> to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with
> libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy
> is included in their hadoop-0.20-native package and it is also installed on
> each of the nodes.
>
> But when I run a mapreduce task with "mapred.map.output.compression.codec"
> set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
>
> java.lang.RuntimeException: native snappy library not available
>
>
> Any idea why?
>
>
>
> Thanks,
> Marek M.
>
>

Hi Marek,

Moving this to [EMAIL PROTECTED] as its CDH specific. I've bcc'd
mapreduce-user@ and cc'd you in case you aren't a subscriber (@
https://groups.google.com/a/cloudera.org/group/cdh-user)

A few questions though:
- What OS are you running and what arch? Check via "lsb_release -a"
and "uname -a"
-- Is your JVM also same as your arch? Check via
"/usr/java/default/bin/java -version"
- Why do you provide
LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64 as
mapred.child.env? It isn't usually required if you have
hadoop-0.20-native installed and the TaskTrackers were restarted since
then.
- Do other codecs (such as GZip and Deflate (Default)) work fine?

On Mon, Jun 11, 2012 at 7:24 PM, Marek Miglinski <[EMAIL PROTECTED]> wrote:
> Hi,
>
> I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy is included in their hadoop-0.20-native package and it is also installed on each of the nodes.
>
> But when I run a mapreduce task with "mapred.map.output.compression.codec" set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
>
> java.lang.RuntimeException: native snappy library not available
>
>
> Any idea why?
>
>
>
> Thanks,
> Marek M.

--
Harsh J