Hi,

I looked through the syslogs and found the following exceptions.
Can anyone help me to figure out the point of error?

java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 22 more
Caused by: java.lang.RuntimeException:
java.lang.NoSuchMethodException:
com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.getNewAggregationBuffer(GenericUDAFBridge.java:160)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.newAggregations(GroupByOperator.java:536)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.initializeOp(GroupByOperator.java:332)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:62)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:133)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:444)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:98)
... 22 more
Caused by: java.lang.NoSuchMethodException:
com.orzota.hive.udaf.groupby.UDAFTopNPercent$TopNPercentEvaluator.<init>()
at java.lang.Class.getConstructor0(Class.java:2706)
at java.lang.Class.getDeclaredConstructor(Class.java:1985)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:109)
... 37 more
2013-02-12 07:21:14,049 INFO org.apache.hadoop.mapred.Task: Runnning
cleanup for the task

Thanks,
Abhishek

From: Abhishek Bhattacharya <[EMAIL PROTECTED]>
Date: Tue, Feb 12, 2013 at 12:48 PM
Subject: Re: Help to solve UDAF errors!
To: [EMAIL PROTECTED]
Hi Mark,

Thanks for the response!
The UDAFPercentile.java have two terminate() methods since it is handling
two different input types by the two inner classes: PercentileLongEvaluator
and PercentileLongArrayEvaluator.
I am handling only a single input type of double from one table column to
the iterate() method and wish to return an ArrayList<DoubleWritable> from
the terminate() method.
What is wrong in my class?
Moreover, is there any way for UDF/UDAF/UDTF which can process all the rows
of the table and output only a subset of the total rows based on some
aggregation function of one column attribute i.e., similar to my case of
computing the top-n-percent of a column attribute and output the entire set
of filtered rows with all other columns from the table?

Thanks,
Abhishek

On Sun, Feb 10, 2013 at 12:36 PM, Mark Grover
<[EMAIL PROTECTED]>wrote:

Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University

Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University

Hello,

This is in reference to the below mentioned UDAF.
As you can notice, I am collecting all the elements in a list for each
mapper. Now, at the reducer I merge all the lists and calculate the top N
percent records to return back from the terminate(). Now, clearly this is
not a scalable solution with a table of trillion records. I am also unable
to make optimal decision at mapper in iterate/terminatePartial so that I
can discard non-eligible elements since top N percent can be optimally
computed at the reducer by collecting all the elements. This is unlike the
top N problem where we are able to take optimal partial decisions.
Any ideas to address this issue regarding optimization/approximation
techniques so that I need not keep all the elements to avoid scalability
issues?

Thanks,
Abhishek

---------- Forwarded message ----------
From: Abhishek Bhattacharya <[EMAIL PROTECTED]>
Date: Wed, Feb 13, 2013 at 7:20 PM
Subject: Re: Help to solve UDAF errors!
To: [EMAIL PROTECTED]
Hi Robin,

Thanks for the response.
The point mentioned by you is one of the many other serious issues in the
code.
I have fixed all of them and it is pretty much in a good shape presently
after some testing.
You can find it at:
https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Let me know if you face any other issues!

Thanks,
Abhishek
On Wed, Feb 13, 2013 at 7:02 PM, Robin Morris <[EMAIL PROTECTED]> wrote:

>  Hi,
>
>  You might well have found the error yourself by now, but if not, the
> problem is that you missed the "static" keyword off the declaration of
> TopNPercentEvaluator.  Line 18 of your code should read
>
> public static class TopNPercentEvaluator implements UDAFEvaluator {
>
>  then this error goes away.
>
>  A new error appears, however, because there's a bug in your terminate()
> function that reads off the end of the array.
>
>  Robin
>
>
>   From: Abhishek Bhattacharya <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Tuesday, February 12, 2013 10:29 AM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Fwd: Help to solve UDAF errors!
>
>    Hi,
>
>  I looked through the syslogs and found the following exceptions.
>  Can anyone help me to figure out the point of error?
>
> java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University

Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University