Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - how may map-reduce needed in a hive query


Copy link to this message
-
how may map-reduce needed in a hive query
Richard 2013-01-23, 03:45
I am wondering how to determine the number of map-reduce for a hive query.
for example, the following query
select
sum(c1),
sum(c2),
k1
from
{
select transform(*) using 'mymapper'  as c1, c2, k1
from t1
} a group by k1;
when i run this query, it takes two map-reduce, but I expect it to take only 1.
in the map stage, using 'mymapper' as the mapper, then shuffle the mapper output by k1 and perform sum reduce in the reducer.
so why hive takes 2 map-reduce?
+
Richard 2013-01-23, 05:54
+
Nitin Pawar 2013-01-23, 06:05
+
Nitin Pawar 2013-01-23, 04:07