Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> native-lzo library not available issue with terasort


Copy link to this message
-
Re: native-lzo library not available issue with terasort
Another approach, arguably a bit better in my opinion, is through the  
hadoop-gpl-compression project (http://code.google.com/p/hadoop-gpl-compression/
). It also incorporates Johan Oskarsson's H-4640 patch. A detailed  
description on how to use it with lzo-less hadoop distribution can be  
found at: http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ.

Thanks, Hong

On Jul 21, 2009, at 8:05 PM, Gross, Danny wrote:

> Thanks Aaron, for the quick response.
>
> Best Regards,
>
> Danny
>
> -----Original Message-----
> From: Aaron Kimball [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, July 21, 2009 9:10 PM
> To: [EMAIL PROTECTED]
> Subject: Re: native-lzo library not available issue with terasort
>
> Native LZO support was removed from Hadoop due to licensing
> restrictions. See
> http://www.cloudera.com/blog/2009/06/24/parallel-lzo-splittable-compression-for-hadoop/
> for a writeup on how to reenable it in your local build.
>
> - Aaron
>
> On Tue, Jul 21, 2009 at 7:02 PM, Gross,  
> Danny<[EMAIL PROTECTED]> wrote:
>> Hello,
>>
>>
>>
>> I've been running terasort on multiple cluster configurations, and
>> attempted to duplicate some of the configuration settings that Yahoo!
>> used for the Minute Sort.
>>
>>
>>
>> In particular, I set the mapred.map.output.compression.codec  
>> property to
>> value "org.apache.hadoop.io.compress.LzoCodec"  in hadoop-
>> site.xml.  I
>> am using hadoop-0.19.1.
>>
>>
>>
>> The teragen program runs fine, and completes with improved time  
>> with my
>> new settings.  However, when I run the terasort program, the  
>> following
>> error is thrown from the map tasks, and the job ultimately fails:
>>
>>
>>
>> "java.lang.RuntimeException: native-lzo library not available at
>> org
>> .apache.hadoop.io.compress.LzoCodec.getCompressorType(LzoCodec.java:1
>> 30) at
>> org
>> .apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:98)
>> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:93) at
>> org.apache.hadoop.mapred.MapTask
>> $MapOutputBuffer.sortAndSpill(MapTask.ja
>> va:961) at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:
>> 842)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at
>> org.apache.hadoop.mapred.Child.main(Child.java:158)"
>>
>>
>>
>> I've searched other places for an answer, and am coming up short.  
>> Any
>> help out there would be greatly appreciated.
>>
>>
>>
>> Best regards,
>>
>>
>>
>> Danny
>>
>>
>>
>> Danny B. Gross
>>
>> Solutions Engineering
>>
>> Spansion,  Inc.
>>
>> email:  [EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>
>>
>>
>>
>>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB