Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> java.lang.RuntimeException: native-lzo library not available


Copy link to this message
-
Re: java.lang.RuntimeException: native-lzo library not available
Sounds like lzo is not set up correctly on slave nodes. Can you check
what's suggested in this conversation?
https://groups.google.com/forum/#!msg/elephantbird-dev/1be_Tjyd2gw/aY1e8w_egTEJ
On Thu, Jul 18, 2013 at 9:27 PM, Bhavesh Shah <[EMAIL PROTECTED]>wrote:

> Hello,
>
> I have written one PIG Script and tried to execute it, but after executing
> some part it gives me error java.io.IOException: Spill failed. I have
> included below statements in my script. And also I have set the classpath
> for hadoop-LZO jar.
> 1) set mapred.compress.map.output true;
> 2) set mapred.map.output.compression.codec
> com.hadoop.compression.lzo.LzopCodec;
>
> This error is caused by java.lang.RuntimeException: native-lzo library not
> available. But I have set the CLASSPATH for Hadoop-LZO jar.
>
> After searching for this error I came to know some like:
> "Since you have a very large number of records, if the individual records
> are small it's likely the
> map task is spilling not because the data buffer is full, but because the
> accounting area is full."
>
> So what should I do to avoid the Spill Failed?
> Below is the exception I got.
>
> Backend error message
> ---------------------
> java.io.IOException: Spill failed
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1213)
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1194)
>  at java.io.DataOutputStream.writeByte(DataOutputStream.java:153)
>  at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:555)
>  at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:435)
>  at
> org.apache.pig.data.utils.SedesHelper.writeGenericTuple(SedesHelper.java:135)
>  at org.apache.pig.data.BinInterSedes.writeTuple(BinInterSedes.java:613)
>  at org.apache.pig.data.BinInterSedes.writeDatum(BinInterSedes.java:443)
>  at org.apache.pig.data.BinSedesTuple.write(BinSedesTuple.java:41)
>  at
> org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:123)
>  at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
>  at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1061)
>  at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
>  at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>  at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
>  at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
>  at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
>  at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>  at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:416)
>  at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>  at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.RuntimeException: native-lzo library not available
>  at
> com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:135)
>  at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:100)
>  at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:112)
>  at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:101)
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1407)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB