Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - how to disable mapred.reduce.tasks


Copy link to this message
-
how to disable mapred.reduce.tasks
Igor Tatarinov 2011-06-29, 23:16
I set mapred.reduce.tasks manually to have a single wave of reducers (does
that make sense, by the way?)

When I save the data, I often end up with a bunch of small files because we
use compression and Hive doesn't seem to merge small compressed files.

So my question is: can I disable mapred.reduce.tasks somehow and make Hive
use the hive.exec.reducers.bytes.per.reducer instead to reduce the number of
output files? It seems the former overrides the latter.