Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Upload files directly to hdfs from standard out using pipes


Copy link to this message
-
Upload files directly to hdfs from standard out using pipes
Hi,
I found out that :
https://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs/hadoopfs.py

can be used to write data directly to HDFS without writing to a local
filesystem but I am not able to understand how.

Could somebody tell me how can I do this ? Or is there some other best way to do it ?

Basiclly my use case is to pipe standard out from a program directly
to HDFS so that it does not have to go through the disk write.
Thanks in advance !

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB