Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> Hive UDAF convert problem


Copy link to this message
-
Hive UDAF convert problem
Hi all.

I'm writing a hive UDAF to calculate page view per session. Java source is blow:

----
public class CalculateAvgPVPerSession extends UDAF {

/**
* @author Cheng Su
*
*/
public static class CountSessionUDAFEvaluator implements UDAFEvaluator {

private VisitSessions visitSessions = new VisitSessions();

/* (non-Javadoc)
* @see org.apache.hadoop.hive.ql.exec.UDAFEvaluator#init()
*/
@Override
public void init() {
// do nothing
}

public boolean iterate(Text value) {
visitSessions.append(value.toString());
return true;
}

public VisitSessions terminatePartial() {
return visitSessions;
}


public boolean merge(VisitSessions other) {
visitSessions.merge(other);
return true;
}


public FloatWritable terminate() {
return new FloatWritable(visitSessions.getAveragePVPerSession());
}
}

}
----

VisitSessions is a class which contains a private field
java.util.TreeMap. Source is blow

----

public class VisitSessions {

private static final DateFormat dateFormat = new
SimpleDateFormat("yyyyMMddHHmmss");

private final long interval;

private static final class Session {
private long start;
private long end;

long getPeriod() {
return end - start;
}
}

private long pageView = 0L;

private TreeMap<Long, Session> sessionMap = Maps.newTreeMap();

// ... do sth ...

       public void merge(VisitSessions other) {
for (final Entry<Long, Session> otherSessionEntry :
other.sessionMap.entrySet()) {
mergeOne(otherSessionEntry.getValue());
}
pageView += other.pageView;
}


}

----
When I use this UDAF, I get this exception :
----
java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row (tag=0)
{"key":{"_col0":0,"_col1":2011,"_col2":10},"value":{"_col0":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}},"_col1":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}}},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:268)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:519)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:420)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row (tag=0)
{"key":{"_col0":0,"_col1":2011,"_col2":10},"value":{"_col0":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}},"_col1":{"interval":1800000,"pageview":8957,"sessionmap":{1319818373000:{"start":1319818373000,"end":1319818731000},1319821763000:{"start":1319821763000,"end":1319824141000},1319858388000:{"start":1319858388000,"end":1319865262000}}}},"alias":0}
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:256)
... 7 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: cannot set field private java.util.TreeMap
com.lietou.datawarehouse.hive.helper.VisitSessions.sessionMap of class
com.lietou.datawarehouse.hive.helper.VisitSessions
com.lietou.datawarehouse.hive.helper.VisitSessions@5f78dc08
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:737)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:247)
... 7 more
Caused by: java.lang.RuntimeException: cannot set field private
java.util.TreeMap
com.lietou.datawarehouse.hive.helper.VisitSessions.sessionMap of class
com.lietou.datawarehouse.hive.helper.VisitSessions
com.lietou.datawarehouse.hive.helper.VisitSessions@5f78dc08
at org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.setStructFieldData(ReflectionStructObjectInspector.java:180)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:240)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFUtils$ConversionHelper.convertIfNecessary(GenericUDFUtils.java:345)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFBridge$GenericUDAFBridgeEvaluator.merge(GenericUDAFBridge.java:176)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:142)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:600)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:724)
... 9 more
Caused by: java.lang.IllegalArgumentException: Can not set
java.util.TreeMap field
com.lietou.datawarehouse.hive.helper.VisitSessions.sessionMap to
java.util.HashMap
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(Unknown
Source)
at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(Unknown
Source)
at sun.reflect.UnsafeObjectFieldAccessorImpl.set(Unknown Source)
at java.lang.reflect.Field.set(Unknown Source)
at org.apache.hadoop.hive.serde2.objectinspector.ReflectionStructObjectInspector.setStructFieldData(ReflectionStructObjectInspector.java:178)
... 16 more

It happened at the reduce phase. Why hive try to set a HashMap to the
TreeMap
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB