Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Skewed Join


Copy link to this message
-
Skewed Join
sumit ghosh 2013-06-07, 06:42
Hi,
 
I am hit by skewed Join, my last reducer is getting same number of Reduce input groups/records.
Reduce input groups                      432,446,942
Reduce shuffle bytes                  13,012,613,275
Reduce input records                     432,446,942 
 
Why is this happening? I have turned on skew join optimization:
hive.optimize.skewjoin=true;
hive.skewjoin.key=100000;
 
 
Thanks,
Sumit