Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive, mail # user - subquery + lateral view fails without count


Copy link to this message
-
Re: subquery + lateral view fails without count
Mark Grover 2012-04-24, 23:39
Hi Ruben,
Looks like pastie is down (http://pastie.org/) because of recent DDOS attacks. Can you please post your queries elsewhere?

Mark

Mark Grover, Business Intelligence Analyst
OANDA Corporation

www: oanda.com www: fxtrade.com
e: [EMAIL PROTECTED]

"Best Trading Platform" - World Finance's Forex Awards 2009.
"The One to Watch" - Treasury Today's Adam Smith Awards 2009.
----- Original Message -----
From: "Ruben de Vries" <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, April 23, 2012 9:08:16 AM
Subject: subquery + lateral view fails without count
It’s a bit of a weird case but I thought I might share it and hopefully find someone who can confirm this to be a bug or tell me I should do things differently!

Here you can find a pastie with the full create and select queries: http://pastie.org/3838924

I’ve got two tables:

`visit_stats` with cols date_int, memberId and parts

`member_map` with cols member_id, gender

I use LATERAL VIEW explode(parts) to explode the parts MAP and group on those and I want to JOIN the member_map onto the visit_stats to group by gender.

Since LATERAL VIEW and JOIN together aren’t supported I’m doing the join in a subquery and then LATERAL VIEW onto that.

It’s working as long as my query contains a COUNT ( memberId ), but if I removed that from the result it will crash with the error below:

java.lang.RuntimeException: Error in configuring object

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)

at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)

at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)

at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:431)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:416)

at org.apache.hadoop.mapred.Child$4.run(Child.java:268)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:396)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)

at org.apache.hadoop.mapred.Child.main(Child.java:262)

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)

... 9 more

Caused by: java.lang.RuntimeException: Reduce operator initialization failed

at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157)

... 14 more

Caused by: java.lang.RuntimeException: cannot find field _col1 from [0:_col4, 1:_col6, 2:_col10]

at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:345)

at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.getStructFieldRef(StandardStructObjectInspector.java:143)

at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)

at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:896)

at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:922)

at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)

at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)

at org.apache.hadoop.hive.ql.exec.JoinOperator.initializeOp(JoinOperator.java:60)

at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)

at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)

... 14 more