Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hive >> mail # user >> How can I import other file in a python transform script


Copy link to this message
-
Re: How can I import other file in a python transform script
hi all:
I have found the answer here
<http://osdir.com/ml/hive-user-hadoop-apache/2010-05/msg00038.html>
:
Adding the following lines before the import
solved the problem:

import sys
import os
sys.path.append(os.getcwd())

2012/5/14 wzc <[EMAIL PROTECTED]>

> Hi all:
> I try to run simple transform script in hive and my script is written in
> python. But when I try to import other file in the script the task fails.
> There may be some basic classes which are used by many transform scripts,
> so I would like to know how to import other file in my transform script.
> Below is my test data:
>
> Hive CLI Query:
> add file /home/hive_test/empty.py;
> add file /home/hive_test/normal.py;
> insert overwrite table test
> select
>   transform(d.*)
>      using './normal.py'
>      as (`name`)
> from
> (
>    select
>      name
>    from test_1
> ) d;
>
> normal.py:
> #!/usr/bin/python
>
> import sys
> import empty
>
> if __name__ == "__main__":
>         for l in sys.stdin:
>                 print l.strip('\n')
>
> empty.py:
> #!/usr/bin/python
> class Empty:
>         pass
>
>
>
> error msg:
>
> java.lang.RuntimeException: Hive Runtime Error while closing operators
> at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hit error while closing ..
> at org.apache.hadoop.hive.ql.exec.ScriptOperator.close(ScriptOperator.java:452)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:566)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)
> ... 8 more
>
>
> Thanks.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB