Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - Upload files directly to hdfs from standard out using pipes


Copy link to this message
-
Upload files directly to hdfs from standard out using pipes
Mapred Learn 2011-01-12, 10:07
Hi,
I found out that :
https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs/hadoopfs.py

can be used to write data directly to HDFS without writing to a local
filesystem but I am not able to understand how.

Could somebody tell me how can I do this ? Or is there some other best way to do it ?

Basiclly my use case is to pipe standard out from a program directly
to HDFS so that it does not have to go through the disk write.
Thanks in advance !