Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >>


Hello,
I am a Newbie to using UDF's on hive. But
implemented these GenericJDF ( https://issues.apache.org/jira/browse/HIVE-2361 ) on hive 0.9.0 and hadoop 1.1.1. Was able to add jar to hive

hive> select * from emp;
OK
1    10    1000
2    10    1200
3    12    1500
4    12    300
5    12    1800
6    20    5000
7    20    7000
8    20    10000
Time taken: 0.191 seconds

hive> add jar /usr/local/Cellar/hive/0.9.0/libexec/lib/GenUDF.jar;                        
Added /usr/local/Cellar/hive/0.9.0/libexec/lib/GenUDF.jar to class path
Added resource: /usr/local/Cellar/hive/0.9.0/libexec/lib/GenUDF.jar

hive> create temporary function nexr_sum as 'com.nexr.platform.analysis.udf.GenericUDFSum';
OK
Time taken: 0.012 seconds

and kicked the sample sql shown below. 
SELECT t.empno, t.deptno, t.sal, nexr_sum(hash(t.deptno),t.sal) as sal_sum
FROM (
select a.empno, a.deptno, a.sal from emp a
distribute by hash(a.deptno)
sort BY a.deptno, a.empno
) t;

The sql failed with errors. Any pointers or advise towards resolving this is much appreciated.

2013-02-13 23:30:18,925 INFO org.apache.hadoop.mapred.JobTracker:
Adding task (REDUCE) 'attempt_201302132324_0002_r_000000_3' to tip
task_201302132324_0002_r_000000, for tracker
'tracker_192.168.0.151:localhost/127.0.0.1:50099'
2013-02-13 23:30:18,925 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201302132324_0002_r_000000_2'
2013-02-13 23:30:26,484 INFO org.apache.hadoop.mapred.TaskInProgress:
Error from attempt_201302132324_0002_r_000000_3:
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:486)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Reduce operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:157)
... 14 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:137)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:896)
at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:922)
at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:60)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:433)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:389)
at org.apache.hadoop.hive.ql.exec.ExtractOperator.initializeOp(ExtractOperator.java:40)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:357)
at org.apache.hadoop.hive.ql.exec.ExecReducer.configure(ExecReducer.java:150)
... 14 more
2013-02-13 23:30:29,819 INFO org.apache.hadoop.mapred.TaskInProgress: TaskInProgress task_201302132324_0002_r_000000 has failed 4 times.
2013-02-13 23:30:29,820 INFO org.apache.hadoop.mapred.JobInProgress: TaskTracker at '192.168.0.151' turned 'flaky'
.... 12 more lines..

Tried different function "GenericUDFMax".. same error.

Any pointers/advise, what could be wrong?
+
Dean Wampler 2013-02-14, 14:41