Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Issue with loading the Snappy Codec

Copy link to this message
Re: Issue with loading the Snappy Codec
You need three things. 1 install snappy to a place the system can pick
it out automatically or add it to your java.library.path

Then add the full name of the codec to io.compression.codecs.

hive> set io.compression.codecs;

On Sun, Apr 15, 2012 at 8:36 AM, Bas Hickendorff
> Hello Jay,
> My input is just a csv file (created it myself), so I am sure it is
> not compressed in any way. Also, the same input works when I use the
> standalone example (using the hadoop executable in the bin folder).
> When I try to integrate it in a larger java program it fails....  :(
> Regards,
> Bas
> On Sun, Apr 15, 2012 at 2:30 PM, JAX <[EMAIL PROTECTED]> wrote:
>> That is odd---- why would it crash when your m/r job did not rely on snappy?
>> One possibility : Maybe because your input is snappy compressed, Hadoop is detecting that compression, and trying to use the snappy codec to decompress.?
>> Jay Vyas
>> On Apr 15, 2012, at 5:08 AM, Bas Hickendorff <[EMAIL PROTECTED]> wrote:
>>> Hello John,
>>> I did restart them (in fact, I did a full reboot of the machine). The
>>> error is still there.
>>> I guess my question is: is it expected that Hadoop needs to do
>>> something with the Snappycodec when mapred.compress.map.output is set
>>> to false?
>>> Regards,
>>> Bas
>>> On Sun, Apr 15, 2012 at 12:04 PM, john smith <[EMAIL PROTECTED]> wrote:
>>>> Can you restart tasktrackers once and run the job again? It refreshes the
>>>> class path.
>>>> On Sun, Apr 15, 2012 at 11:58 AM, Bas Hickendorff
>>>> <[EMAIL PROTECTED]>wrote:
>>>>> Thanks.
>>>>> The native snappy libraries I have installed. However, I use the
>>>>> normal jars that you get when downloading Hadoop, I am not compiling
>>>>> Hadoop myself.
>>>>> I do not want to use the snappy codec (I don't care about compression
>>>>> at the moment), but it seems it is needed anyway? I added this to the
>>>>> mapred-site.xml:
>>>>> <property>
>>>>>        <name>mapred.compress.map.output</name>
>>>>>        <value>false</value>
>>>>> </property>
>>>>> But it still fails with the error of my previous email (SnappyCodec not
>>>>> found).
>>>>> Regards,
>>>>> Bas
>>>>> On Sat, Apr 14, 2012 at 6:30 PM, Vinod Kumar Vavilapalli
>>>>> <[EMAIL PROTECTED]> wrote:
>>>>>> Hadoop has integrated snappy via installed native libraries instead of
>>>>> snappy-java.jar (ref https://issues.apache.org/jira/browse/HADOOP-7206)
>>>>>>  - You need to have the snappy system libraries (snappy and
>>>>> snappy-devel) installed before you compile hadoop. (RPMs are available on
>>>>> the web, http://pkgs.org/centos-5-rhel-5/epel-i386/21/ for example)
>>>>>>  - When you build hadoop, you will need to compile the native
>>>>> libraries(by passing -Dcompile.native=true to ant) to avail snappy support.
>>>>>>  - You also need to make sure that snappy system library is available on
>>>>> the library path for all mapreduce tasks at runtime. Usually if you install
>>>>> them on /usr/lib or /usr/local/lib, it should work.
>>>>>> HTH,
>>>>>> +Vinod
>>>>>> On Apr 14, 2012, at 4:36 AM, Bas Hickendorff wrote:
>>>>>>> Hello,
>>>>>>> When I start a map-reduce job, it starts, and after a short while,
>>>>>>> fails with the error below (SnappyCodec not found).
>>>>>>> I am currently starting the job from other Java code (so the Hadoop
>>>>>>> executable in the bin directory is not used anymore), but in principle
>>>>>>> this seems to work (in the admin of the Jobtracker the job shows up
>>>>>>> when it starts). However after a short while the map task fails with:
>>>>>>> java.lang.IllegalArgumentException: Compression codec