Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> listdir() python function is not wokring on hadoop


Copy link to this message
-
listdir() python function is not wokring on hadoop
Hi all

   is there any one who successfully used listdir() function to retrieve
files one by one from HDFS using python script.
 if __name__ == '__main__':

    for filename in os.listdir("/user/hdmaster/XML2"):
    print filename

ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks
exceeded allowed limit. FailedCount: 1. LastFailedTask:
task_201312020139_0025_m_000000
13/12/02 05:20:50 INFO streaming.StreamJob: killJob...

My intention is to take files one by one to parse.

Any help or suggestion on this will be so much helpful to me

Thanks
Haider
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB