Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> merging the size of the reduce output


Copy link to this message
-
Re: merging the size of the reduce output
Looking at
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java,
hive.merge.mapredfiles is effective if there is a reducer for your job.
Otherwise you should have set hive.merge.mapfiles to true.

On Sat, Jun 12, 2010 at 11:22 PM, Sammy Yu <[EMAIL PROTECTED]> wrote:

> Hi,
>    I'm running the latest version of trunk r953172.  I'm doing doing a
> dynamic partition insert overwrite query which generates a lot of small
> files in each of the partition.  I was hoping this could be solved by
> setting hive.merge.mapredfiles to true.  However, it seems like whenever the
> job is submitted it is always set to false, thus it doesnt seem to have any
> effect.  I also tried to modified this property in the hive-default.xml, but
> it didn't work either.
>
> Thanks,
> Sammy
>
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB