Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - Hive UDAF convert problem


Copy link to this message
-
Hive UDAF convert problem
Cheng Su 2012-11-13, 04:01
Hi all.

I'm writing a hive UDAF to calculate page view per session. Java source is blow:

----
public class CalculateAvgPVPerSession extends UDAF {

/**
* @author Cheng Su
*
*/
public static class CountSessionUDAFEvaluator implements UDAFEvaluator {

private VisitSessions visitSessions = new VisitSessions();

/* (non-Javadoc)
* @see org.apache.hadoop.hive.ql.exec.UDAFEvaluator#init()
*/
@Override
public void init() {
// do nothing
}

public boolean iterate(Text value) {
visitSessions.append(value.toString());
return true;
}

public VisitSessions terminatePartial() {
return visitSessions;
}


public boolean merge(VisitSessions other) {
visitSessions.merge(other);
return true;
}


public FloatWritable terminate() {
return new FloatWritable(visitSessions.getAveragePVPerSession());
}
}

}
----

VisitSessions is a class which contains a private field
java.util.TreeMap. Source is blow

----

public class VisitSessions {

private static final DateFormat dateFormat = new
SimpleDateFormat("yyyyMMddHHmmss");

private final long interval;

private static final class Session {
private long start;
private long end;

long getPeriod() {
return end - start;
}
}

private long pageView = 0L;

private TreeMap<Long, Session> sessionMap = Maps.newTreeMap();

// ... do sth ...

       public void merge(VisitSessions other) {
for (final Entry<Long, Session> otherSessionEntry :
other.sessionMap.entrySet()) {
mergeOne(otherSessionEntry.getValue());
}
pageView += other.pageView;
}


}

----
When I use this UDAF, I get this exception :
----
java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=0)
{"key":{"_col0":0,"_col1":2011,"_col2":10},"value":{"_col0":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}},"_col1":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}}},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row (tag=0)
{"key":{"_col0":0,"_col1":2011,"_col2":10},"value":{"_col0":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}},"_col1":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}}},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
... 7 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot set field private java.util.TreeMap
com.lietou.datawarehouse.hive.helper.VisitSessions.sessionMap of class
com.lietou.datawarehouse.hive.helper.VisitSessions
com.lietou.datawarehouse.hive.helper.VisitSessions@5f78dc08
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:737)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
... 7 more
Caused by: java.lang.RuntimeException: cannot set field private
java.util.TreeMap
com.lietou.datawarehouse.hive.helper.VisitSessions.sessionMap of class
com.lietou.datawarehouse.hive.helper.VisitSessions
com.lietou.datawarehouse.hive.helper.VisitSessions@5f78dc08
at org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.setStructFieldData(ReflectionStructObjectInspector.java:180)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:240)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.convertIfNecessary(GenericUDFUtils.java:345)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:142)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:600)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:724)
... 9 more
Caused by: java.lang.IllegalArgumentException: Can not set
java.util.TreeMap field
com.lietou.datawarehouse.hive.helper.VisitSessions.sessionMap to
java.util.HashMap
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(Unknown
Source)
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(Unknown
Source)
at sun.reflect.UnsafeObjectFieldAccessorImpl.set(Unknown Source)
at java.lang.reflect.Field.set(Unknown Source)
at org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.setStructFieldData(ReflectionStructObjectInspector.java:178)
... 16 more

It happened at the reduce phase. Why hive try to set a HashMap to the
TreeMap
+
Mark Grover 2012-11-15, 07:01
+
Cheng Su 2012-11-15, 08:45