Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive, mail # user - How can I import other file in a python transform script


Copy link to this message
-
How can I import other file in a python transform script
wzc 2012-05-14, 03:08
Hi all:
I try to run simple transform script in hive and my script is written in
python. But when I try to import other file in the script the task fails.
There may be some basic classes which are used by many transform scripts,
so I would like to know how to import other file in my transform script.
Below is my test data:

Hive CLI Query:
add file /home/hive_test/empty.py;
add file /home/hive_test/normal.py;
insert overwrite table test
select
  transform(d.*)
     using './normal.py'
     as (`name`)
from
(
   select
     name
   from test_1
) d;

normal.py:
#!/usr/bin/python

import sys
import empty

if __name__ == "__main__":
        for l in sys.stdin:
                print l.strip('\n')

empty.py:
#!/usr/bin/python
class Empty:
        pass

error msg:

java.lang.RuntimeException: Hive Runtime Error while closing operators
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hit error
while closing ..
at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:452)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
... 8 more
Thanks.
+
wzc 2012-05-19, 04:24