Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - How to make the query compiler not determine the number of reducers?


Copy link to this message
-
Re: How to make the query compiler not determine the number of reducers?
Bejoy KS 2012-04-28, 05:43
Hi Thomas
       Hive automatically sets the number of reducers for you. But you can easily override them at CLI. Before executing your query
hive>SET mapred.reduce.tasks=n;

Where n is the required num of reducers.
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: "Ryabin, Thomas" <[EMAIL PROTECTED]>
Date: Fri, 27 Apr 2012 16:48:25
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: How to make the query compiler not determine the number of reducers?

Hi,

 

When I run a query that uses a custom UDF I made, one of the lines it
prints out is:

Number of reduce tasks determined at compile time: 1

 

And this causes the MapReduce job to have only 1 reducer. Is there a way
to make it so the compiler does not determine the number of reduce tasks
to create, so I can specify the number myself?

 

The query in question is:

select test_udf(name, store) from employees join stores;

 

Thanks,

Thomas