Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Hive UDAF convert problem


+
Cheng Su 2012-11-13, 04:01
+
Mark Grover 2012-11-15, 07:01
Copy link to this message
-
Re: Hive UDAF convert problem
Thanks a lot. I will look into it.

> I don't have the entire code so it's hard for me to say but does it help if
> you change:
>  private TreeMap<Long, Session> sessionMap =...
> to be
>  private Map<Long, Session> sessionMap =...
>

Actually I did change the source like what you suggested to avoid the
convert exception.
But I need to use some TreeMap method, so I had to check and convert
the sessionMap to a TreeMap before I use it.

Thanks a lot.
On Thu, Nov 15, 2012 at 3:01 PM, Mark Grover
<[EMAIL PROTECTED]> wrote:
> Hi Cheng,
> It's reflection that's causing you the problem. You seem to be using the old
> class (UDAF) to implement UDAF. While that may still be fine, just so that
> you know there is a newer, better performing method to implement UDAFs (more
> on that at https://cwiki.apache.org/Hive/genericudafcasestudy.html) that
> doesn't require the use of reflection. I would recommend using creating a
> GenericUDAF over the old method if you are creating a a new UDAF
>
> I don't have the entire code so it's hard for me to say but does it help if
> you change:
>  private TreeMap<Long, Session> sessionMap =...
> to be
>  private Map<Long, Session> sessionMap =...
>
> That would be a good programming practice anyways.
>
> Mark
>
> On Mon, Nov 12, 2012 at 8:01 PM, Cheng Su <[EMAIL PROTECTED]> wrote:
>>
>> Hi all.
>>
>> I'm writing a hive UDAF to calculate page view per session. Java source is
>> blow:
>>
>> ----
>> public class CalculateAvgPVPerSession extends UDAF {
>>
>>         /**
>>          * @author Cheng Su
>>          *
>>          */
>>         public static class CountSessionUDAFEvaluator implements
>> UDAFEvaluator {
>>
>>                 private VisitSessions visitSessions = new VisitSessions();
>>
>>                 /* (non-Javadoc)
>>                  * @see
>> org.apache.hadoop.hive.ql.exec.UDAFEvaluator#init()
>>                  */
>>                 @Override
>>                 public void init() {
>>                         // do nothing
>>                 }
>>
>>                 public boolean iterate(Text value) {
>>                         visitSessions.append(value.toString());
>>                         return true;
>>                 }
>>
>>                 public VisitSessions terminatePartial() {
>>                         return visitSessions;
>>                 }
>>
>>
>>                 public boolean merge(VisitSessions other) {
>>                         visitSessions.merge(other);
>>                         return true;
>>                 }
>>
>>
>>                 public FloatWritable terminate() {
>>                         return new
>> FloatWritable(visitSessions.getAveragePVPerSession());
>>                 }
>>         }
>>
>> }
>> ----
>>
>> VisitSessions is a class which contains a private field
>> java.util.TreeMap. Source is blow
>>
>> ----
>>
>> public class VisitSessions {
>>
>>         private static final DateFormat dateFormat = new
>> SimpleDateFormat("yyyyMMddHHmmss");
>>
>>         private final long interval;
>>
>>         private static final class Session {
>>                 private long start;
>>                 private long end;
>>
>>                 long getPeriod() {
>>                         return end - start;
>>                 }
>>         }
>>
>>         private long pageView = 0L;
>>
>>         private TreeMap<Long, Session> sessionMap = Maps.newTreeMap();
>>
>>         // ... do sth ...
>>
>>        public void merge(VisitSessions other) {
>>                 for (final Entry<Long, Session> otherSessionEntry :
>> other.sessionMap.entrySet()) {
>>                         mergeOne(otherSessionEntry.getValue());
>>                 }
>>                 pageView += other.pageView;
>>         }
>>
>>
>> }
>>
>> ----
>> When I use this UDAF, I get this exception :
>> ----
>> java.lang.RuntimeException:
>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
>> while processing row (tag=0)
>>
>> {"key":{"_col0":0,"_col1":2011,"_col2"

Regards,
Cheng Su
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB