+
Cheng Su 2012-11-13, 04:01
+
Mark Grover 2012-11-15, 07:01
-
Re: Hive UDAF convert problemCheng Su 2012-11-15, 08:45
Thanks a lot. I will look into it.
> I don't have the entire code so it's hard for me to say but does it help if
> you change:
> private TreeMap<Long, Session> sessionMap =...
> to be
> private Map<Long, Session> sessionMap =...
>
Actually I did change the source like what you suggested to avoid the
convert exception.
But I need to use some TreeMap method, so I had to check and convert
the sessionMap to a TreeMap before I use it.
Thanks a lot.
On Thu, Nov 15, 2012 at 3:01 PM, Mark Grover
<[EMAIL PROTECTED]> wrote:
> Hi Cheng,
> It's reflection that's causing you the problem. You seem to be using the old
> class (UDAF) to implement UDAF. While that may still be fine, just so that
> you know there is a newer, better performing method to implement UDAFs (more
> on that at https://cwiki.apache.org/Hive/genericudafcasestudy.html) that
> doesn't require the use of reflection. I would recommend using creating a
> GenericUDAF over the old method if you are creating a a new UDAF
>
> I don't have the entire code so it's hard for me to say but does it help if
> you change:
> private TreeMap<Long, Session> sessionMap =...
> to be
> private Map<Long, Session> sessionMap =...
>
> That would be a good programming practice anyways.
>
> Mark
>
> On Mon, Nov 12, 2012 at 8:01 PM, Cheng Su <[EMAIL PROTECTED]> wrote:
>>
>> Hi all.
>>
>> I'm writing a hive UDAF to calculate page view per session. Java source is
>> blow:
>>
>> ----
>> public class CalculateAvgPVPerSession extends UDAF {
>>
>> /**
>> * @author Cheng Su
>> *
>> */
>> public static class CountSessionUDAFEvaluator implements
>> UDAFEvaluator {
>>
>> private VisitSessions visitSessions = new VisitSessions();
>>
>> /* (non-Javadoc)
>> * @see
>> org.apache.hadoop.hive.ql.exec.UDAFEvaluator#init()
>> */
>> @Override
>> public void init() {
>> // do nothing
>> }
>>
>> public boolean iterate(Text value) {
>> visitSessions.append(value.toString());
>> return true;
>> }
>>
>> public VisitSessions terminatePartial() {
>> return visitSessions;
>> }
>>
>>
>> public boolean merge(VisitSessions other) {
>> visitSessions.merge(other);
>> return true;
>> }
>>
>>
>> public FloatWritable terminate() {
>> return new
>> FloatWritable(visitSessions.getAveragePVPerSession());
>> }
>> }
>>
>> }
>> ----
>>
>> VisitSessions is a class which contains a private field
>> java.util.TreeMap. Source is blow
>>
>> ----
>>
>> public class VisitSessions {
>>
>> private static final DateFormat dateFormat = new
>> SimpleDateFormat("yyyyMMddHHmmss");
>>
>> private final long interval;
>>
>> private static final class Session {
>> private long start;
>> private long end;
>>
>> long getPeriod() {
>> return end - start;
>> }
>> }
>>
>> private long pageView = 0L;
>>
>> private TreeMap<Long, Session> sessionMap = Maps.newTreeMap();
>>
>> // ... do sth ...
>>
>> public void merge(VisitSessions other) {
>> for (final Entry<Long, Session> otherSessionEntry :
>> other.sessionMap.entrySet()) {
>> mergeOne(otherSessionEntry.getValue());
>> }
>> pageView += other.pageView;
>> }
>>
>>
>> }
>>
>> ----
>> When I use this UDAF, I get this exception :
>> ----
>> java.lang.RuntimeException:
>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
>> while processing row (tag=0)
>>
>> {"key":{"_col0":0,"_col1":2011,"_col2"
Regards,
Cheng Su
> I don't have the entire code so it's hard for me to say but does it help if
> you change:
> private TreeMap<Long, Session> sessionMap =...
> to be
> private Map<Long, Session> sessionMap =...
>
Actually I did change the source like what you suggested to avoid the
convert exception.
But I need to use some TreeMap method, so I had to check and convert
the sessionMap to a TreeMap before I use it.
Thanks a lot.
On Thu, Nov 15, 2012 at 3:01 PM, Mark Grover
<[EMAIL PROTECTED]> wrote:
> Hi Cheng,
> It's reflection that's causing you the problem. You seem to be using the old
> class (UDAF) to implement UDAF. While that may still be fine, just so that
> you know there is a newer, better performing method to implement UDAFs (more
> on that at https://cwiki.apache.org/Hive/genericudafcasestudy.html) that
> doesn't require the use of reflection. I would recommend using creating a
> GenericUDAF over the old method if you are creating a a new UDAF
>
> I don't have the entire code so it's hard for me to say but does it help if
> you change:
> private TreeMap<Long, Session> sessionMap =...
> to be
> private Map<Long, Session> sessionMap =...
>
> That would be a good programming practice anyways.
>
> Mark
>
> On Mon, Nov 12, 2012 at 8:01 PM, Cheng Su <[EMAIL PROTECTED]> wrote:
>>
>> Hi all.
>>
>> I'm writing a hive UDAF to calculate page view per session. Java source is
>> blow:
>>
>> ----
>> public class CalculateAvgPVPerSession extends UDAF {
>>
>> /**
>> * @author Cheng Su
>> *
>> */
>> public static class CountSessionUDAFEvaluator implements
>> UDAFEvaluator {
>>
>> private VisitSessions visitSessions = new VisitSessions();
>>
>> /* (non-Javadoc)
>> * @see
>> org.apache.hadoop.hive.ql.exec.UDAFEvaluator#init()
>> */
>> @Override
>> public void init() {
>> // do nothing
>> }
>>
>> public boolean iterate(Text value) {
>> visitSessions.append(value.toString());
>> return true;
>> }
>>
>> public VisitSessions terminatePartial() {
>> return visitSessions;
>> }
>>
>>
>> public boolean merge(VisitSessions other) {
>> visitSessions.merge(other);
>> return true;
>> }
>>
>>
>> public FloatWritable terminate() {
>> return new
>> FloatWritable(visitSessions.getAveragePVPerSession());
>> }
>> }
>>
>> }
>> ----
>>
>> VisitSessions is a class which contains a private field
>> java.util.TreeMap. Source is blow
>>
>> ----
>>
>> public class VisitSessions {
>>
>> private static final DateFormat dateFormat = new
>> SimpleDateFormat("yyyyMMddHHmmss");
>>
>> private final long interval;
>>
>> private static final class Session {
>> private long start;
>> private long end;
>>
>> long getPeriod() {
>> return end - start;
>> }
>> }
>>
>> private long pageView = 0L;
>>
>> private TreeMap<Long, Session> sessionMap = Maps.newTreeMap();
>>
>> // ... do sth ...
>>
>> public void merge(VisitSessions other) {
>> for (final Entry<Long, Session> otherSessionEntry :
>> other.sessionMap.entrySet()) {
>> mergeOne(otherSessionEntry.getValue());
>> }
>> pageView += other.pageView;
>> }
>>
>>
>> }
>>
>> ----
>> When I use this UDAF, I get this exception :
>> ----
>> java.lang.RuntimeException:
>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
>> while processing row (tag=0)
>>
>> {"key":{"_col0":0,"_col1":2011,"_col2"
Regards,
Cheng Su
