Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - How can I load external files within jython UDF?


Copy link to this message
-
How can I load external files within jython UDF?
Young Ng 2012-12-09, 20:52
Hi,

I am trying to load some external resources within my jython udf functions, e.g:

@outputSchema(....)
def test():
    f = open('test.txt.')
    text = f.read()
    f.close()
    return text

I have place the 'test.txt' on both working folder and hdfs, and I got the following error:
   IOError: (2, 'No such file or directory', 'test.txt')

I have also try to print out the working path of jython, with os.getcwd(), below is what I got:
  /home/hduser/tmp/mapred/local/taskTracker/hduser/jobcache/job_201212080111_0007/attempt_201212080111_0007_m_000000_0/work
  ....

I suspect that I can use absolute path within udf, but how can I transfer the external resources to
other hadoop datanodes?
Thanks,
Young Wu
+
John Gordon 2012-12-09, 22:18
+
Young Ng 2012-12-09, 23:18