Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Use of virtual columns in joins


+
Peter Marron 2013-06-10, 08:57
+
Ashutosh Chauhan 2013-06-10, 15:47
+
Peter Marron 2013-06-25, 09:56
Copy link to this message
-
Re: Use of virtual columns in joins
Yes, it's a bug. I've booked on https://issues.apache.org/jira/browse/HIVE-4790.

2013/6/25 Peter Marron <[EMAIL PROTECTED]>:
> Hi,
>
>
>
> Sorry for the delay but I finally got around to testing these queries with
> Hive version 11.
>
> Things are improved. Two of the three queries now run fine. However one
> query still fails.
>
> So this query runs fine:
>
>
>
>                 SELECT *,a.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
> b.rownumber = a.number;
>
> But this one (which is _very_ similar)
>
>
>
>                 SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
> b.rownumber = a.number;
>
> fails with this error:
>
>
>
>     > SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber > a.number;
>
> Automatically selecting local only mode for query
>
> Total MapReduce jobs = 1
>
> setting HADOOP_USER_NAME        pmarron
>
> 13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property
> hive.metastore.local no longer has any effect. Make sure to provide a valid
> value for hive.metastore.uris if you are connecting to a remote metastore.
>
> Execution log at: /tmp/pmarron/.log
>
> 2013-06-25 10:52:56     Starting to launch local task to process map join;
> maximum memory = 932118528
>
> java.lang.RuntimeException: cannot find field block__offset__inside__file
> from [0:rownumber, 1:offset]
>
>         at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
>
>         at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
>
>         at
> org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
>
>         at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
>
>         at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
>
>         at
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
>
>         at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
>
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
>
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
>
>         at
> org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Execution failed with exit status: 2
>
> Obtaining error information
>
>
>
> Task failed!
>
> Task ID:
>
>   Stage-4
>
>
>
> Logs:
>
>
>
> /tmp/pmarron/hive.log
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapredLocalTask
>
>
>
> There really doesn’t seem to be anything helpful in the logs either.
>
> It seems a little weird that it can find the virtual column in the first
> table, but not the second.
>
> Again, these are not blocking me. I’m just reporting these results as they
> may expose a bug.
>
>
>
> Regards,
>
>
>
> Z
>
>
>
> From: Ashutosh Chauhan [mailto:[EMAIL PROTECTED]]
> Sent: 10 June 2013 16:48
> To: [EMAIL PROTECTED]