Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> how to let hive support lzo


Copy link to this message
-
how to let hive support lzo
hi ,all:
     i already install and testing lzo in hadoop and hbase,all success,but
when i try it on hive ,it failed ,how can i do let hive can recognize lzo?
hive> set mapred.map.output.compression.codec;
mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
hive> set
mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec
hive> select count(*) from test;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_1374463239553_0003, Tracking URL http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/>
Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1374463239553_0003
Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 1
2013-07-22 13:33:27,243 Stage-1 map = 0%,  reduce = 0%
2013-07-22 13:33:45,403 Stage-1 map = 100%,  reduce = 0%
Ended Job = job_1374463239553_0003 with errors
Error during job, obtaining debugging information...
Job Tracking URL:
http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/>
Examining task ID: task_1374463239553_0003_m_000000 (and more) from job
job_1374463239553_0003
Task with the most failures(4):
-----
Task ID:
  task_1374463239553_0003_m_000000
URL:

http://CH22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000<http://ch22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000>
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: native-lzo library not available
        at
com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
        at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:104)
        at
org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:118)
        at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:115)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1580)
        at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1457)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB