Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> possible to change replication factor at file creation time (with copyFromLocal)?


+
Julian Bui 2013-05-31, 16:57
Copy link to this message
-
Re: possible to change replication factor at file creation time (with copyFromLocal)?
Hi Julian,

Yes, "dfs" subcommand accepts config overrides via -D. Just do "hadoop
dfs -Ddfs.replication=X -copyFromLocal …".

On Fri, May 31, 2013 at 10:27 PM, Julian Bui <[EMAIL PROTECTED]> wrote:
> Hi hadoop users,
>
> I am aware that you can set the replication factor of a file after it's been
> created, but can you do it as you copy files to the HDFS?  My hope/intuition
> is that if you were able to reduce the replication factor of a file while
> copying, the copy time would decrease.  I'm finding it difficult waiting for
> large data sets to copy over.
>
> I am currently doing:
>
> hadoop dfs -copyFromLocal "/copy/from/path/" input
>
> and am wondering if it's possible to also specify something like -setrep on
> the same line.  -setsrep requires you to specify the file, which implies
> that it has to exist first, requiring two separate commands.
>
> Thanks in advance,
> -Julian

--
Harsh J
+
Julian Bui 2013-05-31, 19:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB