Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - how to let hive support lzo


Copy link to this message
-
Re: how to let hive support lzo
ch huang 2013-07-22, 05:41
# hbase org.apache.hadoop.hbase.util.CompressionTest
hdfs://CH22:9000/alex/my.txt lzo
13/07/22 13:27:58 WARN conf.Configuration: hadoop.native.lib is deprecated.
Instead, use io.native.lib.available
13/07/22 13:27:59 INFO util.ChecksumType: Checksum using
org.apache.hadoop.util.PureJavaCrc32
13/07/22 13:27:59 INFO util.ChecksumType: Checksum can use
org.apache.hadoop.util.PureJavaCrc32C
13/07/22 13:27:59 ERROR metrics.SchemaMetrics: Inconsistent configuration.
Previous configuration for using table name in metrics: true, new
configuration: false
13/07/22 13:27:59 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/07/22 13:27:59 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
13/07/22 13:27:59 INFO compress.CodecPool: Got brand-new compressor
[.lzo_deflate]
13/07/22 13:28:00 INFO compress.CodecPool: Got brand-new decompressor
[.lzo_deflate]
SUCCESS
# hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.15.jar
com.hadoop.compression.lzo.LzoIndexer /alex
13/07/22 09:39:04 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
13/07/22 09:39:04 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
13/07/22 09:39:04 INFO lzo.LzoIndexer: LZO Indexing directory /alex...
13/07/22 09:39:04 INFO lzo.LzoIndexer:   LZO Indexing directory
hdfs://CH22:9000/alex/alex_t...
13/07/22 09:39:04 INFO lzo.LzoIndexer:   [INDEX] LZO Indexing file
hdfs://CH22:9000/alex/sqoop-1.99.2-bin-hadoop200.tar.gz.lzo, size 0.02 GB...
13/07/22 09:39:05 WARN conf.Configuration: hadoop.native.lib is deprecated.
Instead, use io.native.lib.available
13/07/22 09:39:06 INFO lzo.LzoIndexer:   Completed LZO Indexing in 1.16
seconds (13.99 MB/s).  Index size is 0.52 KB.

13/07/22 09:39:06 INFO lzo.LzoIndexer:   [INDEX] LZO Indexing file
hdfs://CH22:9000/alex/test1.lzo, size 0.00 GB...
13/07/22 09:39:06 INFO lzo.LzoIndexer:   Completed LZO Indexing in 0.08
seconds (0.00 MB/s).  Index size is 0.01 KB.
On Mon, Jul 22, 2013 at 1:37 PM, ch huang <[EMAIL PROTECTED]> wrote:

> hi ,all:
>      i already install and testing lzo in hadoop and hbase,all success,but
> when i try it on hive ,it failed ,how can i do let hive can recognize lzo?
>
>
> hive> set mapred.map.output.compression.codec;
>
> mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec
> hive> set
> mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec
> hive> select count(*) from test;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks determined at compile time: 1
> In order to change the average load for a reducer (in bytes):
>   set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
>   set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
>   set mapred.reduce.tasks=<number>
> Starting Job = job_1374463239553_0003, Tracking URL > http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/>
> Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1374463239553_0003
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 1
> 2013-07-22 13:33:27,243 Stage-1 map = 0%,  reduce = 0%
> 2013-07-22 13:33:45,403 Stage-1 map = 100%,  reduce = 0%
> Ended Job = job_1374463239553_0003 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: http://CH22:8088/proxy/application_1374463239553_0003/<http://ch22:8088/proxy/application_1374463239553_0003/>
> Examining task ID: task_1374463239553_0003_m_000000 (and more) from job
> job_1374463239553_0003
> Task with the most failures(4):
> -----
> Task ID:
>   task_1374463239553_0003_m_000000
> URL:
>
> http://CH22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000<http://ch22:8088/taskdetails.jsp?jobid=job_1374463239553_0003&tipid=task_1374463239553_0003_m_000000>