Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Symbolic links available in 1.0.3?


Copy link to this message
-
Symbolic links available in 1.0.3?
Is it possible to use symbolic links in 1.0.3?

 

If yes: can I use symbolic links to create a single, final directory
structure of files from many locations; then use DistCp/S3DistCp to copy
that final directory structure to another filesystem such as S3?

 

Usecase:

I currently launch 4 S3DistCp jobs concurrently (a royal pain in the a** to
do) and copy files from 4 different locations to various subfolders of a
single directory on S3 (the final result directory).

 

I'd love to create 1 S3DistCp job to do all of that work, but
S3DistCp/DistCp don't support specifying multiple source/destinations.

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB