Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - is it possible to disable running termiatePartial and merge() methods in UDAF


Copy link to this message
-
is it possible to disable running termiatePartial and merge() methods in UDAF
java8964 java8964 2012-10-01, 16:16

Hi,
I am trying to implement a UDAF of Kurtosis (<�a href="http://en.wikipedia.org/wiki/Kurtosis">http://en.wikipedia.org/wiki/Kurtosis<�/a> in the hive.
I already found a library to do it, from Apache commons math (<�a href="http://commons.apache.org/math/apidocs/org/apache/commons/math/stat/descriptive/moment/Kurtosis.html">http://commons.apache.org/math/apidocs/org/apache/commons/math/stat/descriptive/moment/Kurtosis.html<�/a>).
But it does NOT support merging partial result in it.
I am not a Math guru, so I don't know if this Kurtosis can be done in parallel or not.
My question is simple, if I use the above library to implement this UDAF in hive, is there a configuration in the hive (either at runtime or at UDAF level) to make surethat for this UDAF I implemented, the terminatePartial/merge will NEVER be invoked?
I have 2 settings I think maybe will do the trick:
1) set hive.exec.reducers.max=12) set hive.map.aggr=false
Does either one will do the thing I want? Or anything else? Or is it even possible?
Thanks
Yong