Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> executing linux command from hadoop (python)


Copy link to this message
-
executing linux command from hadoop (python)
Hi,
 Lets say that I have a data which interacts with a rest api like

%curl hostname data

Now, I have the following script:

#!/usr/bin/env python
import sys,os

cmd = """curl http://localhost  --data  '"""
string = " "
for line in sys.stdin:
    line = line.rstrip(os.linesep)
    string += line

os.system(cmd + string+"'")
Now, if i give a sample file for data, and run the above script with

cat data.txt | python mapper.py

It works perfectly. But will this work if i execute on hadoop as well?
I am trying to set up hadoop on local mode to check it out but I think it
will take me sometime to get there?
Any experiences, suggestions?
Thanks
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB