-Re: Want query to use more reducers
Sean Busbey 2013-09-30, 19:28
It sounds like you should tweak the settings for how Hive handles query
1) Tune the guessed number of reducers based on input size
Defaults to 1G. Based on your description, it sounds like this is probably
still at default.
In this case, you should also set a max # of reducers based on your cluster
I usually set this to the # reduce slots, if there's a decent chance I'll
get to saturate the cluster. If not, don't worry about it.
2) Hard code a number of reducers
Setting this will cause Hive to always use that number. It defaults to -1,
which tells hive to use the heuristic about input size to guess.
In either of the above cases, you should look at the options to merge small
files (search for "merge" in the configuration property list) to avoid
getting lots of little outputs.
On Mon, Sep 30, 2013 at 11:31 AM, Keith Wiley <[EMAIL PROTECTED]> wrote:
> I have a query that doesn't use reducers as efficiently as I would hope.
> If I run it on a large table, it uses more reducers, even saturating the
> cluster, as I desire. However, on smaller tables it uses as low as a
> single reducer. While I understand there is a logic in this (not using
> multiple reducers until the data size is larger), it is nevertheless
> inefficient to run a query for thirty minutes leaving the entire cluster
> vacant when the query could distribute the work evenly and wrap things up
> in a fraction of the time. The query is shown below (abstracted to its
> basic form). As you can see, it is a little atypical: it is a nested query
> which obviously implies two map-reduce jobs and it uses a script for the
> reducer stage that I am trying to speed up. I thought the "distribute by"
> clause should make it use the reducers more evenly, but as I said, that is
> not the behavior I am seeing.
> Any ideas how I could improve this situation?
> CREATE TABLE output_table ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' as
> SELECT * FROM (
> FROM (
> SELECT * FROM input_table
> DISTRIBUTE BY input_column_1 SORT BY input_column_1 ASC,
> input_column_2 ASC, input_column_etc ASC) q
> SELECT TRANSFORM(*)
> USING 'python my_reducer_script.py' AS(
> ) s
> ORDER BY output_column_1;
> Keith Wiley [EMAIL PROTECTED] keithwiley.com
> "Luminous beings are we, not this crude matter."
> -- Yoda