Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - Help to solve UDAF errors!


Copy link to this message
-
Re: Help to solve UDAF errors!
Abhishek Bhattacharya 2013-02-10, 07:08
Thanks for the response.
The link to the code is:
https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Please let me know to fix it!

Thanks,
Abhishek

On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover <[EMAIL PROTECTED]>wrote:

> Abhishek,
> The code doesn't seem to be complete.
>
> Look at
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor reference. It has two terminate()'s - one for UDAF and one for the
> Evaluator.
>
> Do you mind posting your complete code on github somewhere so it's easier
> to analyze?
>
> Mark
>
> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> I have implemented a simple UDAF for top-n-percent as follows:
>> import java.util.ArrayList;
>> import java.util.Collections;
>>
>> import org.apache.hadoop.hive.ql.exec.UDAF;
>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>>
>> public class UDAFTopNPercent extends UDAF{
>>
>>     public static class Result {
>>         ArrayList<Double> list;
>>         double min;
>>     }
>>
>>     public class TopNPercentEvaluator implements UDAFEvaluator {
>>
>>         private Result res;
>>         private int rowIndex;
>>         private int percent;
>>
>>         public TopNPercentEvaluator() {
>>             super();
>>             res = new Result();
>>             init();
>>             rowIndex = 0;
>>         }
>>         @Override
>>         public void init() {
>>             res.list = new ArrayList<Double>();
>>             res.min = Double.MAX_VALUE;
>>         }
>>
>>         public boolean iterate(Double rowVal, int pct) {
>>             ArrayList<Double> resList = res.list;
>>             rowIndex++;
>>             resList.add(rowVal);
>>             percent = pct;
>>             return true;
>>         }
>>
>>         public ArrayList<Double> terminatePartial() {
>>             ArrayList<Double> resList = res.list;
>>             Collections.sort(resList);
>>             return resList;
>>         }
>>
>>         public boolean merge(ArrayList<Double> otherList) {
>>             ArrayList<Double> resList = res.list;
>>             resList.addAll(otherList);
>>             return true;
>>         }
>>
>>         public ArrayList<Double> terminate() {
>>             ArrayList<Double> resList = res.list;
>>             double num_rows = (double)percent/100.0*rowIndex;
>>             Collections.sort(resList);
>>             int lastIdx = resList.size()- (int) num_rows;
>>             if(lastIdx <= 0) {
>>                 return resList;
>>             }
>>             for(int i=0; i<lastIdx; i++) {
>>                 resList.remove(i);
>>             }
>>             return resList;
>>         }
>>     }
>>
>>     /**
>>      * @param args
>>      */
>>     public static void main(String[] args) {
>>         // TODO Auto-generated method stub
>>
>>     }
>>
>> }
>>
>> But throws some error such as first few lines are:
>> FAILED: Hive Internal Error:
>> java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>> cannot be cast to
>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
>> java.lang.ClassCastException:
>> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>> cannot be cast to
>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
>>         at
>> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)
>>         at
>> org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300)
>>         at
>> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)
>>
>> Please help me to debug this!
>> Is it throwing from returning ArrayList<Double> in terminate()?
>> How should I return a List from UDAF?
>>
>> Thanks,
>> Abhishek
>>
>
Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University