Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> snappy codec


+
Marek Miglinski 2012-06-11, 13:54
+
GUOJUN Zhu 2012-06-11, 14:12
Copy link to this message
-
Re: snappy codec
use java lib path

On Mon, Jun 11, 2012 at 7:12 AM, GUOJUN Zhu <[EMAIL PROTECTED]>wrote:

>
> I believe that the mapred.child.env only concerns the spawned child
> process where the map is run.  You probably need to put this path into the
> system property where the map task demon runs and possibly where the job
> controller runs.
>
> Zhu, Guojun
> Modeling Sr Graduate
> 571-3824370
> [EMAIL PROTECTED]
> Financial Engineering
> Freddie Mac
>
>
>      *Marek Miglinski <[EMAIL PROTECTED]>*
>
>    06/11/2012 09:54 AM
>      Please respond to
> [EMAIL PROTECTED]
>
>    To
> "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> cc
>   Subject
> snappy codec
>
>
>
>
> Hi,
>
> I have a Clouderas CDH3u3 installed on my cluster and mapred.child.env set
> to "LD_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64" (with
> libsnappy.so in the folder) in mapred-site.xml. Cloudera says that Snappy
> is included in their hadoop-0.20-native package and it is also installed on
> each of the nodes.
>
> But when I run a mapreduce task with "mapred.map.output.compression.codec"
> set to "org.apache.hadoop.io.compress.SnappyCodec" I get an exception:
>
> java.lang.RuntimeException: native snappy library not available
>
>
> Any idea why?
>
>
>
> Thanks,
> Marek M.
>
>
+
Harsh J 2012-06-11, 14:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB