Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> lzo error


Copy link to this message
-
Re: lzo error
sorry,I already got the problem and address it, sorry to trouble with you!

Best Regards

Malone
2012-06-11

yingnan.ma

发件人: yingnan.ma
发送时间: 2012-06-11  21:57:15
收件人: user
抄送:
主题: lzo error
 
Hi
I already install the lzo, and from the log it sould like successful
mapred.compress.map.output true
&&
syslog logs
2012-06-11 21:50:39,749 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2012-06-11 21:50:40,247 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId2012-06-11 21:50:40,352 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.data.dir;  Ignoring.
2012-06-11 21:50:40,361 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.name.dir;  Ignoring.
2012-06-11 21:50:41,118 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 150
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: data buffer = 119537664/149422080
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: record buffer = 393216/491520
2012-06-11 21:50:41,276 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2012-06-11 21:50:41,280 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
but the pig will give me an error
2012-06-11 21:50:31,507 [Thread-4] ERROR com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native gpl libraryjava.lang.UnsatisfiedLinkError: no gplcompression in java.library.path        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)        at java.lang.Runtime.loadLibrary0(Runtime.java:823)        at java.lang.System.loadLibrary(System.java:1028)        at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)        at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)        at java.lang.Class.forName0(Native Method)        at java.lang.Class.forName(Class.java:247)        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109)        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156)        at org.apache.hadoop.mapreduce.lib.input.
 TextInputFormat.isSplitable(TextInputFormat.java:51)        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:254)        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)        at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:396)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)        at org.apache.hadoop.mapred.JobCl
 ient.submitJob(JobClient.java:807)        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)        at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)        at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)        at java.lang.Thread.run(Thread.java:662)2012-06-11 21:50:31,508 [Thread-4] ERROR com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without native-hadoopAnd the job could run successful, I donot know the reason. Thank you for your help.Best RegardsMalone
2012-06-11