Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> subquery + lateral view fails without count


Copy link to this message
-
Re: subquery + lateral view fails without count
Hi Ruben,
Looks like pastie is down (http://pastie.org/) because of recent DDOS attacks. Can you please post your queries elsewhere?

Mark

Mark Grover, Business Intelligence Analyst
OANDA Corporation

www: oanda.com www: fxtrade.com
e: [EMAIL PROTECTED]

"Best Trading Platform" - World Finance's Forex Awards 2009.
"The One to Watch" - Treasury Today's Adam Smith Awards 2009.
----- Original Message -----
From: "Ruben de Vries" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, April 23, 2012 9:08:16 AM
Subject: subquery + lateral view fails without count
It’s a bit of a weird case but I thought I might share it and hopefully find someone who can confirm this to be a bug or tell me I should do things differently!

Here you can find a pastie with the full create and select queries: http://pastie.org/3838924

I’ve got two tables:

`visit_stats` with cols date_int, memberId and parts

`member_map` with cols member_id, gender

I use LATERAL VIEW explode(parts) to explode the parts MAP and group on those and I want to JOIN the member_map onto the visit_stats to group by gender.

Since LATERAL VIEW and JOIN together aren’t supported I’m doing the join in a subquery and then LATERAL VIEW onto that.

It’s working as long as my query contains a COUNT ( memberId ), but if I removed that from the result it will crash with the error below:

java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:431)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

... 9 more

Caused by: java.lang.RuntimeException: Reduce operator initialization failed

at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157)

... 14 more

Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:_col4, 1:_col6, 2:_col10]

at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:345)

at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:143)

at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)

at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:896)

at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:922)

at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)

at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)

at org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:60)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)

... 14 more
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB