Julian Bui 2013-05-31, 16:57
-Re: possible to change replication factor at file creation time (with copyFromLocal)?
Harsh J 2013-05-31, 17:03
Yes, "dfs" subcommand accepts config overrides via -D. Just do "hadoop
dfs -Ddfs.replication=X -copyFromLocal …".
On Fri, May 31, 2013 at 10:27 PM, Julian Bui <[EMAIL PROTECTED]> wrote:
> Hi hadoop users,
> I am aware that you can set the replication factor of a file after it's been
> created, but can you do it as you copy files to the HDFS? My hope/intuition
> is that if you were able to reduce the replication factor of a file while
> copying, the copy time would decrease. I'm finding it difficult waiting for
> large data sets to copy over.
> I am currently doing:
> hadoop dfs -copyFromLocal "/copy/from/path/" input
> and am wondering if it's possible to also specify something like -setrep on
> the same line. -setsrep requires you to specify the file, which implies
> that it has to exist first, requiring two separate commands.
> Thanks in advance,
Julian Bui 2013-05-31, 19:27