Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Skewed Join


Hi,
 
I am hit by skewed Join, my last reducer is getting same number of Reduce input groups/records.
Reduce input groups                      432,446,942
Reduce shuffle bytes                  13,012,613,275
Reduce input records                     432,446,942 
 
Why is this happening? I have turned on skew join optimization:
hive.optimize.skewjoin=true;
hive.skewjoin.key=100000;
 
 
Thanks,
Sumit 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB