Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hive >> mail # user >> How can I import other file in a python transform script


+
wzc 2012-05-14, 03:08
Copy link to this message
-
Re: How can I import other file in a python transform script
hi all:
I have found the answer here
<http://osdir.com/ml/hive-user-hadoop-apache/2010-05/msg00038.html>
:
Adding the following lines before the import
solved the problem:

import sys
import os
sys.path.append(os.getcwd())

2012/5/14 wzc <[EMAIL PROTECTED]>

> Hi all:
> I try to run simple transform script in hive and my script is written in
> python. But when I try to import other file in the script the task fails.
> There may be some basic classes which are used by many transform scripts,
> so I would like to know how to import other file in my transform script.
> Below is my test data:
>
> Hive CLI Query:
> add file /home/hive_test/empty.py;
> add file /home/hive_test/normal.py;
> insert overwrite table test
> select
>   transform(d.*)
>      using './normal.py'
>      as (`name`)
> from
> (
>    select
>      name
>    from test_1
> ) d;
>
> normal.py:
> #!/usr/bin/python
>
> import sys
> import empty
>
> if __name__ == "__main__":
>         for l in sys.stdin:
>                 print l.strip('\n')
>
> empty.py:
> #!/usr/bin/python
> class Empty:
>         pass
>
>
>
> error msg:
>
> java.lang.RuntimeException: Hive Runtime Error while closing operators
> at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hit error while closing ..
> at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:452)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
> ... 8 more
>
>
> Thanks.
>