Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Fwd: Help to solve UDAF errors!


Copy link to this message
-
Fwd: Help to solve UDAF errors!
Hello,

This is in reference to the below mentioned UDAF.
As you can notice, I am collecting all the elements in a list for each
mapper. Now, at the reducer I merge all the lists and calculate the top N
percent records to return back from the terminate(). Now, clearly this is
not a scalable solution with a table of trillion records. I am also unable
to make optimal decision at mapper in iterate/terminatePartial so that I
can discard non-eligible elements since top N percent can be optimally
computed at the reducer by collecting all the elements. This is unlike the
top N problem where we are able to take optimal partial decisions.
Any ideas to address this issue regarding optimization/approximation
techniques so that I need not keep all the elements to avoid scalability
issues?

Thanks,
Abhishek

---------- Forwarded message ----------
From: Abhishek Bhattacharya <[EMAIL PROTECTED]>
Date: Wed, Feb 13, 2013 at 7:20 PM
Subject: Re: Help to solve UDAF errors!
To: [EMAIL PROTECTED]
Hi Robin,

Thanks for the response.
The point mentioned by you is one of the many other serious issues in the
code.
I have fixed all of them and it is pretty much in a good shape presently
after some testing.
You can find it at:
https://github.com/Abhishek2301/Hive/blob/master/src/UDAFTopNPercent.java
Let me know if you face any other issues!

Thanks,
Abhishek
On Wed, Feb 13, 2013 at 7:02 PM, Robin Morris <[EMAIL PROTECTED]> wrote:

>  Hi,
>
>  You might well have found the error yourself by now, but if not, the
> problem is that you missed the "static" keyword off the declaration of
> TopNPercentEvaluator.  Line 18 of your code should read
>
> public static class TopNPercentEvaluator implements UDAFEvaluator {
>
>  then this error goes away.
>
>  A new error appears, however, because there's a bug in your terminate()
> function that reads off the end of the array.
>
>  Robin
>
>
>   From: Abhishek Bhattacharya <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Tuesday, February 12, 2013 10:29 AM
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Fwd: Help to solve UDAF errors!
>
>    Hi,
>
>  I looked through the syslogs and found the following exceptions.
>  Can anyone help me to figure out the point of error?
>
> java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:387)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> at org.apache.hadoop.mapred.Child.main(Child.java:264)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University

Thanks and Regards,

Abhishek Bhattacharya
PhD Computer Science
School of Computing and Information Sciences
Florida International University