Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Issue with loading the Snappy Codec


Copy link to this message
-
Re: Issue with loading the Snappy Codec
You need three things. 1 install snappy to a place the system can pick
it out automatically or add it to your java.library.path

Then add the full name of the codec to io.compression.codecs.

hive> set io.compression.codecs;
io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec

Edward
On Sun, Apr 15, 2012 at 8:36 AM, Bas Hickendorff
<[EMAIL PROTECTED]> wrote:
> Hello Jay,
>
> My input is just a csv file (created it myself), so I am sure it is
> not compressed in any way. Also, the same input works when I use the
> standalone example (using the hadoop executable in the bin folder).
> When I try to integrate it in a larger java program it fails....  :(
>
> Regards,
>
> Bas
>
> On Sun, Apr 15, 2012 at 2:30 PM, JAX <[EMAIL PROTECTED]> wrote:
>> That is odd---- why would it crash when your m/r job did not rely on snappy?
>>
>> One possibility : Maybe because your input is snappy compressed, Hadoop is detecting that compression, and trying to use the snappy codec to decompress.?
>>
>> Jay Vyas
>> MMSB
>> UCHC
>>
>> On Apr 15, 2012, at 5:08 AM, Bas Hickendorff <[EMAIL PROTECTED]> wrote:
>>
>>> Hello John,
>>>
>>> I did restart them (in fact, I did a full reboot of the machine). The
>>> error is still there.
>>>
>>> I guess my question is: is it expected that Hadoop needs to do
>>> something with the Snappycodec when mapred.compress.map.output is set
>>> to false?
>>>
>>> Regards,
>>>
>>> Bas
>>>
>>> On Sun, Apr 15, 2012 at 12:04 PM, john smith <[EMAIL PROTECTED]> wrote:
>>>> Can you restart tasktrackers once and run the job again? It refreshes the
>>>> class path.
>>>>
>>>> On Sun, Apr 15, 2012 at 11:58 AM, Bas Hickendorff
>>>> <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> Thanks.
>>>>>
>>>>> The native snappy libraries I have installed. However, I use the
>>>>> normal jars that you get when downloading Hadoop, I am not compiling
>>>>> Hadoop myself.
>>>>>
>>>>> I do not want to use the snappy codec (I don't care about compression
>>>>> at the moment), but it seems it is needed anyway? I added this to the
>>>>> mapred-site.xml:
>>>>>
>>>>> <property>
>>>>>        <name>mapred.compress.map.output</name>
>>>>>        <value>false</value>
>>>>> </property>
>>>>>
>>>>> But it still fails with the error of my previous email (SnappyCodec not
>>>>> found).
>>>>>
>>>>> Regards,
>>>>>
>>>>> Bas
>>>>>
>>>>>
>>>>> On Sat, Apr 14, 2012 at 6:30 PM, Vinod Kumar Vavilapalli
>>>>> <[EMAIL PROTECTED]> wrote:
>>>>>>
>>>>>> Hadoop has integrated snappy via installed native libraries instead of
>>>>> snappy-java.jar (ref https://issues.apache.org/jira/browse/HADOOP-7206)
>>>>>>  - You need to have the snappy system libraries (snappy and
>>>>> snappy-devel) installed before you compile hadoop. (RPMs are available on
>>>>> the web, http://pkgs.org/centos-5-rhel-5/epel-i386/21/ for example)
>>>>>>  - When you build hadoop, you will need to compile the native
>>>>> libraries(by passing -Dcompile.native=true to ant) to avail snappy support.
>>>>>>  - You also need to make sure that snappy system library is available on
>>>>> the library path for all mapreduce tasks at runtime. Usually if you install
>>>>> them on /usr/lib or /usr/local/lib, it should work.
>>>>>>
>>>>>> HTH,
>>>>>> +Vinod
>>>>>>
>>>>>> On Apr 14, 2012, at 4:36 AM, Bas Hickendorff wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> When I start a map-reduce job, it starts, and after a short while,
>>>>>>> fails with the error below (SnappyCodec not found).
>>>>>>>
>>>>>>> I am currently starting the job from other Java code (so the Hadoop
>>>>>>> executable in the bin directory is not used anymore), but in principle
>>>>>>> this seems to work (in the admin of the Jobtracker the job shows up
>>>>>>> when it starts). However after a short while the map task fails with:
>>>>>>>
>>>>>>>
>>>>>>> java.lang.IllegalArgumentException: Compression codec
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB