Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> only one mapper


+
闫昆 2013-08-21, 09:17
Copy link to this message
-
Re: only one mapper
LZO files are combinable so check your max split setting.
http://mail-archives.apache.org/mod_mbox/hive-user/201107.mbox/%[EMAIL PROTECTED]%3E

igor
decide.com

On Wed, Aug 21, 2013 at 2:17 AM, 闫昆 <[EMAIL PROTECTED]> wrote:

> hi all when i use hive
> hive job make only one mapper actually my file split 18 block my block
> size is 128MB and data size 2GB
> i use lzo compression and create file.lzo and make index file.lzo.index
> i use hive 0.10.0
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Cannot run job locally: Input Size (= 2304560827) is larger than
> hive.exec.mode.local.auto.inputbytes.max (= 134217728)
> Starting Job = job_1377071515613_0003, Tracking URL > http://hydra0001:8088/proxy/application_1377071515613_0003/
> Kill Command = /opt/module/hadoop-2.0.0-cdh4.3.0/bin/hadoop job  -kill
> job_1377071515613_0003
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2013-08-21 16:44:30,237 Stage-1 map = 0%,  reduce = 0%
> 2013-08-21 16:44:40,495 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU
> 6.81 sec
> 2013-08-21 16:44:41,710 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU
> 6.81 sec
> 2013-08-21 16:44:42,919 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU
> 6.81 sec
> 2013-08-21 16:44:44,117 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU
> 9.95 sec
> 2013-08-21 16:44:45,333 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU
> 9.95 sec
> 2013-08-21 16:44:46,530 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU
> 13.0 sec
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> [EMAIL PROTECTED]
>
>
+
Edward Capriolo 2013-08-21, 17:43
+
pandees waran 2013-08-21, 18:01
+
Sanjay Subramanian 2013-08-21, 19:13
+
闫昆 2013-08-22, 01:41
+
Rajesh Balamohan 2013-08-22, 02:59
+
闫昆 2013-08-22, 03:32
+
Rajesh Balamohan 2013-08-22, 03:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB