Hive, mail # user - Re: Help to solve UDAF errors!

Help to solve UDAF errors!

Abhishek Bhattacharya 2013-02-08, 22:05

Hi,

I have implemented a simple UDAF for top-n-percent as follows:
import java.util.ArrayList;
import java.util.Collections;

import org.apache.hadoop.hive.ql.exec.UDAF;
import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;

public class UDAFTopNPercent extends UDAF{

   public static class Result {
   ArrayList<Double> list;
   double min;
   }

   public class TopNPercentEvaluator implements UDAFEvaluator {

   private Result res;
   private int rowIndex;
   private int percent;

   public TopNPercentEvaluator() {
   super();
   res = new Result();
   init();
   rowIndex = 0;
   }
   @Override
   public void init() {
   res.list = new ArrayList<Double>();
   res.min = Double.MAX_VALUE;
   }

   public boolean iterate(Double rowVal, int pct) {
   ArrayList<Double> resList = res.list;
   rowIndex++;
   resList.add(rowVal);
   percent = pct;
   return true;
   }

   public ArrayList<Double> terminatePartial() {
   ArrayList<Double> resList = res.list;
   Collections.sort(resList);
   return resList;
   }

   public boolean merge(ArrayList<Double> otherList) {
   ArrayList<Double> resList = res.list;
   resList.addAll(otherList);
   return true;
   }

   public ArrayList<Double> terminate() {
   ArrayList<Double> resList = res.list;
   double num_rows = (double)percent/100.0*rowIndex;
   Collections.sort(resList);
   int lastIdx = resList.size()- (int) num_rows;
   if(lastIdx <= 0) {
   return resList;
   }
   for(int i=0; i<lastIdx; i++) {
   resList.remove(i);
   }
   return resList;
   }
   }

   /**
   * @param args
   */
   public static void main(String[] args) {
   // TODO Auto-generated method stub

   }

}

But throws some error such as first few lines are:
FAILED: Hive Internal Error:
java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
cannot be cast to
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
   at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)
   at
org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300)
   at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)

Please help me to debug this!
Is it throwing from returning ArrayList<Double> in terminate()?
How should I return a List from UDAF?

Thanks,
Abhishek

Re: Help to solve UDAF errors!

Mark Grover 2013-02-08, 23:02

Abhishek,
The code doesn't seem to be complete.

Look at
https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor
reference. It has two terminate()'s - one for UDAF and one for the
Evaluator.

Do you mind posting your complete code on github somewhere so it's easier
to analyze?

Mark

On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I have implemented a simple UDAF for top-n-percent as follows:
> import java.util.ArrayList;
> import java.util.Collections;
>
> import org.apache.hadoop.hive.ql.exec.UDAF;
> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>
> public class UDAFTopNPercent extends UDAF{
>
> public static class Result {
> ArrayList<Double> list;
> double min;
> }
>
> public class TopNPercentEvaluator implements UDAFEvaluator {
>
> private Result res;
> private int rowIndex;
> private int percent;
>
> public TopNPercentEvaluator() {
> super();
> res = new Result();
> init();
> rowIndex = 0;
> }
> @Override
> public void init() {
> res.list = new ArrayList<Double>();
> res.min = Double.MAX_VALUE;
> }
>
> public boolean iterate(Double rowVal, int pct) {
> ArrayList<Double> resList = res.list;
> rowIndex++;
> resList.add(rowVal);
> percent = pct;
> return true;
> }
>
> public ArrayList<Double> terminatePartial() {
> ArrayList<Double> resList = res.list;
> Collections.sort(resList);
> return resList;
> }
>
> public boolean merge(ArrayList<Double> otherList) {
> ArrayList<Double> resList = res.list;
> resList.addAll(otherList);
> return true;
> }
>
> public ArrayList<Double> terminate() {
> ArrayList<Double> resList = res.list;
> double num_rows = (double)percent/100.0*rowIndex;
> Collections.sort(resList);
> int lastIdx = resList.size()- (int) num_rows;
> if(lastIdx <= 0) {
> return resList;
> }
> for(int i=0; i<lastIdx; i++) {
> resList.remove(i);
> }
> return resList;
> }
> }
>
> /**
> * @param args
> */
> public static void main(String[] args) {
> // TODO Auto-generated method stub
>
> }
>
> }
>
> But throws some error such as first few lines are:
> FAILED: Hive Internal Error:
> java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)
>
> Please help me to debug this!
> Is it throwing from returning ArrayList<Double> in terminate()?
> How should I return a List from UDAF?
>
> Thanks,
> Abhishek
>

Re: Help to solve UDAF errors!

Abhishek Bhattacharya 2013-02-10, 07:08

Thanks for the response.
The link to the code is:
https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Please let me know to fix it!

Thanks,
Abhishek

On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover <[EMAIL PROTECTED]>wrote:

> Abhishek,
> The code doesn't seem to be complete.
>
> Look at
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor reference. It has two terminate()'s - one for UDAF and one for the
> Evaluator.
>
> Do you mind posting your complete code on github somewhere so it's easier
> to analyze?
>
> Mark
>
> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> I have implemented a simple UDAF for top-n-percent as follows:
>> import java.util.ArrayList;
>> import java.util.Collections;
>>
>> import org.apache.hadoop.hive.ql.exec.UDAF;
>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>>
>> public class UDAFTopNPercent extends UDAF{
>>
>> public static class Result {
>> ArrayList<Double> list;
>> double min;
>> }
>>
>> public class TopNPercentEvaluator implements UDAFEvaluator {
>>
>> private Result res;
>> private int rowIndex;
>> private int percent;
>>
>> public TopNPercentEvaluator() {
>> super();
>> res = new Result();
>> init();
>> rowIndex = 0;
>> }
>> @Override
>> public void init() {
>> res.list = new ArrayList<Double>();
>> res.min = Double.MAX_VALUE;
>> }
>>
>> public boolean iterate(Double rowVal, int pct) {
>> ArrayList<Double> resList = res.list;
>> rowIndex++;
>> resList.add(rowVal);
>> percent = pct;
>> return true;
>> }
>>
>> public ArrayList<Double> terminatePartial() {
>> ArrayList<Double> resList = res.list;
>> Collections.sort(resList);
>> return resList;
>> }
>>
>> public boolean merge(ArrayList<Double> otherList) {
>> ArrayList<Double> resList = res.list;
>> resList.addAll(otherList);
>> return true;
>> }
>>
>> public ArrayList<Double> terminate() {
>> ArrayList<Double> resList = res.list;
>> double num_rows = (double)percent/100.0*rowIndex;
>> Collections.sort(resList);
>> int lastIdx = resList.size()- (int) num_rows;
>> if(lastIdx <= 0) {
>> return resList;
>> }
>> for(int i=0; i<lastIdx; i++) {
>> resList.remove(i);
>> }
>> return resList;
>> }
>> }
>>
>> /**
>> * @param args
>> */
>> public static void main(String[] args) {
>> // TODO Auto-generated method stub
>>
>> }
>>
>> }
>>
>> But throws some error such as first few lines are:
>> FAILED: Hive Internal Error:
>> java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>> cannot be cast to
>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
>> java.lang.ClassCastException:
>> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>> cannot be cast to
>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
>> at
>> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)
>> at
>> org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.<init>(GenericUDFUtils.java:300)
>> at
>> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.init(GenericUDAFBridge.java:129)
>>
>> Please help me to debug this!
>> Is it throwing from returning ArrayList<Double> in terminate()?
>> How should I return a List from UDAF?
>>
>> Thanks,
>> Abhishek
>>
>
Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University

Re: Help to solve UDAF errors!

Mark Grover 2013-02-10, 18:36

Hi Abhishek,
The code looks incomplete.

See the comment at
https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22
Those are all the methods your UDAF class needs to implement but you seem
to be missing them.

Mark

On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:

> Thanks for the response.
> The link to the code is:
> https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
> Please let me know to fix it!
>
> Thanks,
> Abhishek
>
>
>
> On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover <[EMAIL PROTECTED]>wrote:
>
>> Abhishek,
>> The code doesn't seem to be complete.
>>
>> Look at
>> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor reference. It has two terminate()'s - one for UDAF and one for the
>> Evaluator.
>>
>> Do you mind posting your complete code on github somewhere so it's easier
>> to analyze?
>>
>> Mark
>>
>> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:
>>
>>> Hi,
>>>
>>> I have implemented a simple UDAF for top-n-percent as follows:
>>> import java.util.ArrayList;
>>> import java.util.Collections;
>>>
>>> import org.apache.hadoop.hive.ql.exec.UDAF;
>>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>>>
>>> public class UDAFTopNPercent extends UDAF{
>>>
>>> public static class Result {
>>> ArrayList<Double> list;
>>> double min;
>>> }
>>>
>>> public class TopNPercentEvaluator implements UDAFEvaluator {
>>>
>>> private Result res;
>>> private int rowIndex;
>>> private int percent;
>>>
>>> public TopNPercentEvaluator() {
>>> super();
>>> res = new Result();
>>> init();
>>> rowIndex = 0;
>>> }
>>> @Override
>>> public void init() {
>>> res.list = new ArrayList<Double>();
>>> res.min = Double.MAX_VALUE;
>>> }
>>>
>>> public boolean iterate(Double rowVal, int pct) {
>>> ArrayList<Double> resList = res.list;
>>> rowIndex++;
>>> resList.add(rowVal);
>>> percent = pct;
>>> return true;
>>> }
>>>
>>> public ArrayList<Double> terminatePartial() {
>>> ArrayList<Double> resList = res.list;
>>> Collections.sort(resList);
>>> return resList;
>>> }
>>>
>>> public boolean merge(ArrayList<Double> otherList) {
>>> ArrayList<Double> resList = res.list;
>>> resList.addAll(otherList);
>>> return true;
>>> }
>>>
>>> public ArrayList<Double> terminate() {
>>> ArrayList<Double> resList = res.list;
>>> double num_rows = (double)percent/100.0*rowIndex;
>>> Collections.sort(resList);
>>> int lastIdx = resList.size()- (int) num_rows;
>>> if(lastIdx <= 0) {
>>> return resList;
>>> }
>>> for(int i=0; i<lastIdx; i++) {
>>> resList.remove(i);
>>> }
>>> return resList;
>>> }
>>> }
>>>
>>> /**
>>> * @param args
>>> */
>>> public static void main(String[] args) {
>>> // TODO Auto-generated method stub
>>>
>>> }
>>>
>>> }
>>>
>>> But throws some error such as first few lines are:
>>> FAILED: Hive Internal Error:
>>> java.lang.ClassCastException(org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>>> cannot be cast to
>>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector)
>>> java.lang.ClassCastException:
>>> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableFloatObjectInspector
>>> cannot be cast to
>>> org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
>>> at
>>> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConverter(ObjectInspectorConverters.java:116)

Re: Help to solve UDAF errors!

Abhishek Bhattacharya 2013-02-11, 03:47

Hi Mark,

Sorry for the preliminary questions since I am a beginner for Hive.
I have read in both books (Hadoop Definitive Guide and Programming Hive)
that we need to implement the functions: init(), iterate(),
terminatePartial(), merge() and terminate for extending a UDAF and
UDAFEvaluator class.
Even I implemented a group sum function before with the above methods and
it worked fine.
But the link you sent asks to implement init(), aggregate() and evaluate()
which I find it completely new since I used evaluate() for UDF.
Is this some new version of Hive?
The problem here is, I am not able to return a ArrayList of doubles from
the final terminate() function but I have seen other working UDAF's where
the ArrayList of different types can be returned.
So how do I return a ArrayList of Doubles?

Thanks,
Abhishek
On Sun, Feb 10, 2013 at 12:36 PM, Mark Grover
<[EMAIL PROTECTED]>wrote:

> Hi Abhishek,
> The code looks incomplete.
>
> See the comment at
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22
> Those are all the methods your UDAF class needs to implement but you seem
> to be missing them.
>
> Mark
>
> On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:
>
>> Thanks for the response.
>> The link to the code is:
>> https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
>> Please let me know to fix it!
>>
>> Thanks,
>> Abhishek
>>
>>
>>
>> On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover <[EMAIL PROTECTED]>wrote:
>>
>>> Abhishek,
>>> The code doesn't seem to be complete.
>>>
>>> Look at
>>> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor reference. It has two terminate()'s - one for UDAF and one for the
>>> Evaluator.
>>>
>>> Do you mind posting your complete code on github somewhere so it's
>>> easier to analyze?
>>>
>>> Mark
>>>
>>> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi,
>>>>
>>>> I have implemented a simple UDAF for top-n-percent as follows:
>>>> import java.util.ArrayList;
>>>> import java.util.Collections;
>>>>
>>>> import org.apache.hadoop.hive.ql.exec.UDAF;
>>>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>>>>
>>>> public class UDAFTopNPercent extends UDAF{
>>>>
>>>> public static class Result {
>>>> ArrayList<Double> list;
>>>> double min;
>>>> }
>>>>
>>>> public class TopNPercentEvaluator implements UDAFEvaluator {
>>>>
>>>> private Result res;
>>>> private int rowIndex;
>>>> private int percent;
>>>>
>>>> public TopNPercentEvaluator() {
>>>> super();
>>>> res = new Result();
>>>> init();
>>>> rowIndex = 0;
>>>> }
>>>> @Override
>>>> public void init() {
>>>> res.list = new ArrayList<Double>();
>>>> res.min = Double.MAX_VALUE;
>>>> }
>>>>
>>>> public boolean iterate(Double rowVal, int pct) {
>>>> ArrayList<Double> resList = res.list;
>>>> rowIndex++;
>>>> resList.add(rowVal);
>>>> percent = pct;
>>>> return true;
>>>> }
>>>>
>>>> public ArrayList<Double> terminatePartial() {
>>>> ArrayList<Double> resList = res.list;
>>>> Collections.sort(resList);
>>>> return resList;
>>>> }
>>>>
>>>> public boolean merge(ArrayList<Double> otherList) {
>>>> ArrayList<Double> resList = res.list;
>>>> resList.addAll(otherList);
>>>> return true;
>>>> }
>>>>
>>>> public ArrayList<Double> terminate() {
>>>> ArrayList<Double> resList = res.list;
>>>> double num_rows = (double)percent/100.0*rowIndex;
>>>> Collections.sort(resList);
>>>> int lastIdx = resList.size()- (int) num_rows;
>>>> if(lastIdx <= 0) {
Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University

Re: Help to solve UDAF errors!

Abhishek Bhattacharya 2013-02-12, 17:48

Hi Mark,

Thanks for the response!
The UDAFPercentile.java have two terminate() methods since it is handling
two different input types by the two inner classes: PercentileLongEvaluator
and PercentileLongArrayEvaluator.
I am handling only a single input type of double from one table column to
the iterate() method and wish to return an ArrayList<DoubleWritable> from
the terminate() method.
What is wrong in my class?
Moreover, is there any way for UDF/UDAF/UDTF which can process all the rows
of the table and output only a subset of the total rows based on some
aggregation function of one column attribute i.e., similar to my case of
computing the top-n-percent of a column attribute and output the entire set
of filtered rows with all other columns from the table?

Thanks,
Abhishek

On Sun, Feb 10, 2013 at 12:36 PM, Mark Grover
<[EMAIL PROTECTED]>wrote:

> Hi Abhishek,
> The code looks incomplete.
>
> See the comment at
> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java#L22
> Those are all the methods your UDAF class needs to implement but you seem
> to be missing them.
>
> Mark
>
> On Sat, Feb 9, 2013 at 11:08 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:
>
>> Thanks for the response.
>> The link to the code is:
>> https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
>> Please let me know to fix it!
>>
>> Thanks,
>> Abhishek
>>
>>
>>
>> On Fri, Feb 8, 2013 at 5:02 PM, Mark Grover <[EMAIL PROTECTED]>wrote:
>>
>>> Abhishek,
>>> The code doesn't seem to be complete.
>>>
>>> Look at
>>> https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.javafor reference. It has two terminate()'s - one for UDAF and one for the
>>> Evaluator.
>>>
>>> Do you mind posting your complete code on github somewhere so it's
>>> easier to analyze?
>>>
>>> Mark
>>>
>>> On Fri, Feb 8, 2013 at 2:05 PM, Abhishek Bhattacharya <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi,
>>>>
>>>> I have implemented a simple UDAF for top-n-percent as follows:
>>>> import java.util.ArrayList;
>>>> import java.util.Collections;
>>>>
>>>> import org.apache.hadoop.hive.ql.exec.UDAF;
>>>> import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
>>>>
>>>> public class UDAFTopNPercent extends UDAF{
>>>>
>>>> public static class Result {
>>>> ArrayList<Double> list;
>>>> double min;
>>>> }
>>>>
>>>> public class TopNPercentEvaluator implements UDAFEvaluator {
>>>>
>>>> private Result res;
>>>> private int rowIndex;
>>>> private int percent;
>>>>
>>>> public TopNPercentEvaluator() {
>>>> super();
>>>> res = new Result();
>>>> init();
>>>> rowIndex = 0;
>>>> }
>>>> @Override
>>>> public void init() {
>>>> res.list = new ArrayList<Double>();
>>>> res.min = Double.MAX_VALUE;
>>>> }
>>>>
>>>> public boolean iterate(Double rowVal, int pct) {
>>>> ArrayList<Double> resList = res.list;
>>>> rowIndex++;
>>>> resList.add(rowVal);
>>>> percent = pct;
>>>> return true;
>>>> }
>>>>
>>>> public ArrayList<Double> terminatePartial() {
>>>> ArrayList<Double> resList = res.list;
>>>> Collections.sort(resList);
>>>> return resList;
>>>> }
>>>>
>>>> public boolean merge(ArrayList<Double> otherList) {
>>>> ArrayList<Double> resList = res.list;
>>>> resList.addAll(otherList);
>>>> return true;
>>>> }
>>>>
>>>> public ArrayList<Double> terminate() {
>>>> ArrayList<Double> resList = res.list;
>>>> double num_rows = (double)percent/100.0*rowIndex;
>>>> Collections.sort(resList);
>>>> int lastIdx = resList.size()- (int) num_rows;
>>>> if(lastIdx <= 0) {
>>>> return resList;
Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University

Re: Help to solve UDAF errors!

Robin Morris 2013-02-14, 01:02

Hi,

You might well have found the error yourself by now, but if not, the problem is that you missed the "static" keyword off the declaration of TopNPercentEvaluator. Line 18 of your code should read

   public static class TopNPercentEvaluator implements UDAFEvaluator {

then this error goes away.

A new error appears, however, because there's a bug in your terminate() function that reads off the end of the array.

Robin
From: Abhishek Bhattacharya <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Tuesday, February 12, 2013 10:29 AM
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Fwd: Help to solve UDAF errors!

Hi,

I looked through the syslogs and found the following exceptions.
Can anyone help me to figure out the point of error?
java.lang.RuntimeException: Error in configuring object
   at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
   at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
   at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
   at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
   at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
   at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
   ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
   at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
   at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
   at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
   at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
   ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
   ... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
   at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
   ... 22 more
Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
   at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
   at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.getNewAggregationBuffer(GenericUDAFBridge.java:160)
   at org.apache.hadoop.hive.ql.exec.GroupByOperator.newAggregations(GroupByOperator.java:536)
   at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:332)
   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
   at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
   at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:62)
   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
   at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
   at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)
   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
   at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)
   at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
   at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
   ... 22 more
Caused by: java.lang.NoSuchMethodException: com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
   at java.lang.Class.getConstructor0(Class.java:2706)
   at java.lang.Class.getDeclaredConstructor(Class.java:1985)
   at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
   ... 37 more
2013-02-12 07:21:14,049 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
________________________________
Thanks,
Abhishek

From: Abhishek Bhattacharya <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Tue, Feb 12, 2013 at 12:48 PM
Subject: Re: Help to solve UDAF errors!
To: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Hi Mark,

Thanks for the response!
The UDAFPercentile.java have two terminate() methods since it is handling two different input types by the two inner classes: PercentileLongEvaluator and PercentileLongArrayEvaluator.
I am handling only a single input type of double from one table column to the iterate() method and wish to return an ArrayList<DoubleWritable> from the

Re: Help to solve UDAF errors!

Abhishek Bhattacharya 2013-02-14, 01:20

Hi Robin,

Thanks for the response.
The point mentioned by you is one of the many other serious issues in the
code.
I have fixed all of them and it is pretty much in a good shape presently
after some testing.
You can find it at:
https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Let me know if you face any other issues!

Thanks,
Abhishek
On Wed, Feb 13, 2013 at 7:02 PM, Robin Morris <[EMAIL PROTECTED]> wrote:

> Hi,
>
> You might well have found the error yourself by now, but if not, the
> problem is that you missed the "static" keyword off the declaration of
> TopNPercentEvaluator. Line 18 of your code should read
>
> public static class TopNPercentEvaluator implements UDAFEvaluator {
>
> then this error goes away.
>
> A new error appears, however, because there's a bug in your terminate()
> function that reads off the end of the array.
>
> Robin
>
>
> From: Abhishek Bhattacharya <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Tuesday, February 12, 2013 10:29 AM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Fwd: Help to solve UDAF errors!
>
> Hi,
>
> I looked through the syslogs and found the following exceptions.
> Can anyone help me to figure out the point of error?
>
> java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
> ... 22 more
> Caused by: java.lang.RuntimeException: java.lang.NoSuchMethodException: com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
> at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.getNewAggregationBuffer(GenericUDAFBridge.java:160)
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.newAggregations(GroupByOperator.java:536)
> at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:332)

Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University