Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> executing linux command from hadoop (python)

Copy link to this message
executing linux command from hadoop (python)
 Lets say that I have a data which interacts with a rest api like

%curl hostname data

Now, I have the following script:

#!/usr/bin/env python
import sys,os

cmd = """curl http://localhost  --data  '"""
string = " "
for line in sys.stdin:
    line = line.rstrip(os.linesep)
    string += line

os.system(cmd + string+"'")
Now, if i give a sample file for data, and run the above script with

cat data.txt | python mapper.py

It works perfectly. But will this work if i execute on hadoop as well?
I am trying to set up hadoop on local mode to check it out but I think it
will take me sometime to get there?
Any experiences, suggestions?