Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> listdir() python function is not wokring on hadoop


Copy link to this message
-
listdir() python function is not wokring on hadoop
Hi all

   is there any one who successfully used listdir() function to retrieve
files one by one from HDFS using python script.
 if __name__ == '__main__':

    for filename in os.listdir("/user/hdmaster/XML2"):
    print filename

ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks
exceeded allowed limit. FailedCount: 1. LastFailedTask:
task_201312020139_0025_m_000000
13/12/02 05:20:50 INFO streaming.StreamJob: killJob...

My intention is to take files one by one to parse.

Any help or suggestion on this will be so much helpful to me

Thanks
Haider
+
Yigitbasi, Nezih 2013-12-05, 22:50
+
Haider 2013-12-06, 06:12
+
shashwat shriparv 2013-12-06, 08:52
+
Nitin Pawar 2013-12-06, 08:57
+
Yigitbasi, Nezih 2013-12-07, 02:49
+
Haider 2013-12-07, 03:18
+
Nitin Pawar 2013-12-07, 06:30
+
Haider 2013-12-07, 07:19