Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> How to make the query compiler not determine the number of reducers?


Copy link to this message
-
Re: How to make the query compiler not determine the number of reducers?
Hi Thomas
       Hive automatically sets the number of reducers for you. But you can easily override them at CLI. Before executing your query
hive>SET mapred.reduce.tasks=n;

Where n is the required num of reducers.
Regards
Bejoy KS

Sent from handheld, please excuse typos.

-----Original Message-----
From: "Ryabin, Thomas" <[EMAIL PROTECTED]>
Date: Fri, 27 Apr 2012 16:48:25
To: <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: How to make the query compiler not determine the number of reducers?

Hi,

 

When I run a query that uses a custom UDF I made, one of the lines it
prints out is:

Number of reduce tasks determined at compile time: 1

 

And this causes the MapReduce job to have only 1 reducer. Is there a way
to make it so the compiler does not determine the number of reduce tasks
to create, so I can specify the number myself?

 

The query in question is:

select test_udf(name, store) from employees join stores;

 

Thanks,

Thomas
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB