Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> Use of virtual columns in joins


+
Peter Marron 2013-06-10, 08:57
+
Ashutosh Chauhan 2013-06-10, 15:47
+
Peter Marron 2013-06-25, 09:56
Copy link to this message
-
Re: Use of virtual columns in joins
Yes, it's a bug. I've booked on https://issues.apache.org/jira/browse/HIVE-4790.

2013/6/25 Peter Marron <[EMAIL PROTECTED]>:
> Hi,
>
>
>
> Sorry for the delay but I finally got around to testing these queries with
> Hive version 11.
>
> Things are improved. Two of the three queries now run fine. However one
> query still fails.
>
> So this query runs fine:
>
>
>
>                 SELECT *,a.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
> b.rownumber = a.number;
>
> But this one (which is _very_ similar)
>
>
>
>                 SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
> b.rownumber = a.number;
>
> fails with this error:
>
>
>
>     > SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber > a.number;
>
> Automatically selecting local only mode for query
>
> Total MapReduce jobs = 1
>
> setting HADOOP_USER_NAME        pmarron
>
> 13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property
> hive.metastore.local no longer has any effect. Make sure to provide a valid
> value for hive.metastore.uris if you are connecting to a remote metastore.
>
> Execution log at: /tmp/pmarron/.log
>
> 2013-06-25 10:52:56     Starting to launch local task to process map join;
> maximum memory = 932118528
>
> java.lang.RuntimeException: cannot find field block__offset__inside__file
> from [0:rownumber, 1:offset]
>
>         at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
>
>         at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
>
>         at
> org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
>
>         at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
>
>         at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
>
>         at
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
>
>         at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
>
>         at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
>
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
>
>         at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
>
>         at
> org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Execution failed with exit status: 2
>
> Obtaining error information
>
>
>
> Task failed!
>
> Task ID:
>
>   Stage-4
>
>
>
> Logs:
>
>
>
> /tmp/pmarron/hive.log
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapredLocalTask
>
>
>
> There really doesn’t seem to be anything helpful in the logs either.
>
> It seems a little weird that it can find the virtual column in the first
> table, but not the second.
>
> Again, these are not blocking me. I’m just reporting these results as they
> may expose a bug.
>
>
>
> Regards,
>
>
>
> Z
>
>
>
> From: Ashutosh Chauhan [mailto:[EMAIL PROTECTED]]
> Sent: 10 June 2013 16:48
> To: [EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB