Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> is it possible to disable running termiatePartial and merge() methods in UDAF


Copy link to this message
-
is it possible to disable running termiatePartial and merge() methods in UDAF

Hi,
I am trying to implement a UDAF of Kurtosis (<�a href="http://en.wikipedia.org/wiki/Kurtosis">http://en.wikipedia.org/wiki/Kurtosis<�/a> in the hive.
I already found a library to do it, from Apache commons math (<�a href="http://commons.apache.org/math/apidocs/org/apache/commons/math/stat/descriptive/moment/Kurtosis.html">http://commons.apache.org/math/apidocs/org/apache/commons/math/stat/descriptive/moment/Kurtosis.html<�/a>).
But it does NOT support merging partial result in it.
I am not a Math guru, so I don't know if this Kurtosis can be done in parallel or not.
My question is simple, if I use the above library to implement this UDAF in hive, is there a configuration in the hive (either at runtime or at UDAF level) to make surethat for this UDAF I implemented, the terminatePartial/merge will NEVER be invoked?
I have 2 settings I think maybe will do the trick:
1) set hive.exec.reducers.max=12) set hive.map.aggr=false
Does either one will do the thing I want? Or anything else? Or is it even possible?
Thanks
Yong    
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB