Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - problem with hadoop-snappy


+
alxsss@... 2012-11-06, 01:10
+
Binglin Chang 2012-11-06, 02:06
+
alxsss@... 2012-11-06, 21:30
+
Binglin Chang 2012-11-07, 01:34
+
alxsss@... 2012-11-07, 01:42
Copy link to this message
-
Re: problem with hadoop-snappy
Arpit Gupta 2012-11-07, 01:55
Alex

You need to make sure libsnappy.so is available in hadoop's java library path.

The way hadoop sets the java library path is that based on the jdk it will add either 32bit or 64bit libraries.

They need to be present in $HADOOP_HOME/lib/native/Linux-amd64-64 for 64bit jdk or $HADOOP_HOME/lib/native/Linux-i386-32 for 32 bit jdk. So create the libsnappy.so link in the appropriate directories on all your nodes restart your tasktrackers and submit your job again.

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Nov 6, 2012, at 5:42 PM, [EMAIL PROTECTED] wrote:

> Yes, I have installed snappy-1.0.5.
>
> Here is the output of
>  ls -l /usr/local/lib
> total 1468
> lrwxrwxrwx. 1 root root     52 Nov  5 16:00 libjvm.so -> /usr/java/jdk1.6.0_37/jre/lib/amd64/server/libjvm.so
> -rw-r--r--. 1 root root 946358 Nov  3 17:44 liblzo2.a
> -rwxr-xr-x. 1 root root    866 Nov  3 17:44 liblzo2.la
> -rw-r--r--. 1 root root 371174 Nov  5 15:18 libsnappy.a
> -rwxr-xr-x. 1 root root    957 Nov  5 15:18 libsnappy.la
> lrwxrwxrwx. 1 root root     18 Nov  5 15:18 libsnappy.so -> libsnappy.so.1.1.3
> lrwxrwxrwx. 1 root root     18 Nov  5 15:18 libsnappy.so.1 -> libsnappy.so.1.1.3
> -rwxr-xr-x. 1 root root 169393 Nov  5 15:18 libsnappy.so.1.1.3
>
>
>
> -----Original Message-----
> From: Binglin Chang <[EMAIL PROTECTED]>
> To: user <[EMAIL PROTECTED]>
> Sent: Tue, Nov 6, 2012 5:34 pm
> Subject: Re: problem with hadoop-snappy
>
> Have you installed snappy library (which should have libsnappy.so in /usr/lib or /usr/local/lib) first?
>
>
>
> On Wed, Nov 7, 2012 at 5:30 AM, <[EMAIL PROTECTED]> wrote:
> When I used hadoop-1.0.4  without the installation of hadoop-snappy I saw in the logs
>
>  WARN snappy.LoadSnappy: Snappy native library not loaded
>
> So, it was my understanding that snappy is not included with hadoop-1.0.4. After removing the libs I installed, I see again the same warning.
>
> Any ideas how to fix this issue?
>
> Thanks.
> Alex.
>
>
>
>
> -----Original Message-----
> From: Binglin Chang <[EMAIL PROTECTED]>
> To: user <[EMAIL PROTECTED]>
> Sent: Mon, Nov 5, 2012 6:07 pm
> Subject: Re: problem with hadoop-snappy
>
> I think hadoop-1.0.4 already have snappy included, you should not using other third party libraries.
>  
>
>
> On Tue, Nov 6, 2012 at 9:10 AM, <[EMAIL PROTECTED]> wrote:
> Hello,
>
> I use hadoop-1.0.4  I have followed  instruction to install hadoop-snappy at
> http://code.google.com/p/hadoop-snappy/
>
> When I run a mapred job I see
>
> FATAL org.apache.hadoop.mapred.TaskTracker: Task: attempt_201211051656_0002_m_000000_3 - Killed : org.apache.hadoop.io.compress.snappy.SnappyCompressor.compressBytesDirect()I
>
> in task tracker log and the job fails.
>
> Any ideas how to fix this issue.
>
> Thanks in advance.
> Alex.
>
>
>

+
Colin McCabe 2012-11-07, 18:56