Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - merging the size of the reduce output


Copy link to this message
-
Re: merging the size of the reduce output
Ted Yu 2010-06-13, 14:39
Looking at
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java,
hive.merge.mapredfiles is effective if there is a reducer for your job.
Otherwise you should have set hive.merge.mapfiles to true.

On Sat, Jun 12, 2010 at 11:22 PM, Sammy Yu <[EMAIL PROTECTED]> wrote:

> Hi,
>    I'm running the latest version of trunk r953172.  I'm doing doing a
> dynamic partition insert overwrite query which generates a lot of small
> files in each of the partition.  I was hoping this could be solved by
> setting hive.merge.mapredfiles to true.  However, it seems like whenever the
> job is submitted it is always set to false, thus it doesnt seem to have any
> effect.  I also tried to modified this property in the hive-default.xml, but
> it didn't work either.
>
> Thanks,
> Sammy
>
>
>