Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> lzo error


Copy link to this message
-
Re: lzo error
sorry,I already got the problem and address it, sorry to trouble with you!

Best Regards

Malone
2012-06-11

yingnan.ma

发件人: yingnan.ma
发送时间: 2012-06-11  21:57:15
收件人: user
抄送:
主题: lzo error
 
Hi
I already install the lzo, and from the log it sould like successful
mapred.compress.map.output true
&&
syslog logs
2012-06-11 21:50:39,749 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2012-06-11 21:50:40,247 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId2012-06-11 21:50:40,352 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.data.dir;  Ignoring.
2012-06-11 21:50:40,361 WARN org.apache.hadoop.conf.Configuration: /hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a attempt to override final parameter: dfs.name.dir;  Ignoring.
2012-06-11 21:50:41,118 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 150
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: data buffer = 119537664/149422080
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: record buffer = 393216/491520
2012-06-11 21:50:41,276 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl library
2012-06-11 21:50:41,280 INFO com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
but the pig will give me an error
2012-06-11 21:50:31,507 [Thread-4] ERROR com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native gpl libraryjava.lang.UnsatisfiedLinkError: no gplcompression in java.library.path        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738)        at java.lang.Runtime.loadLibrary0(Runtime.java:823)        at java.lang.System.loadLibrary(System.java:1028)        at com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)        at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)        at java.lang.Class.forName0(Native Method)        at java.lang.Class.forName(Class.java:247)        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109)        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156)        at org.apache.hadoop.mapreduce.lib.input.
 TextInputFormat.isSplitable(TextInputFormat.java:51)        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:254)        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)        at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)        at java.security.AccessController.doPrivileged(Native Method)        at javax.security.auth.Subject.doAs(Subject.java:396)        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)        at org.apache.hadoop.mapred.JobCl
 ient.submitJob(JobClient.java:807)        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)        at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)        at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)        at java.lang.Thread.run(Thread.java:662)2012-06-11 21:50:31,508 [Thread-4] ERROR com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without native-hadoopAnd the job could run successful, I donot know the reason. Thank you for your help.Best RegardsMalone
2012-06-11
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB