Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - possible to change replication factor at file creation time (with copyFromLocal)?


+
Julian Bui 2013-05-31, 16:57
Copy link to this message
-
Re: possible to change replication factor at file creation time (with copyFromLocal)?
Harsh J 2013-05-31, 17:03
Hi Julian,

Yes, "dfs" subcommand accepts config overrides via -D. Just do "hadoop
dfs -Ddfs.replication=X -copyFromLocal …".

On Fri, May 31, 2013 at 10:27 PM, Julian Bui <[EMAIL PROTECTED]> wrote:
> Hi hadoop users,
>
> I am aware that you can set the replication factor of a file after it's been
> created, but can you do it as you copy files to the HDFS?  My hope/intuition
> is that if you were able to reduce the replication factor of a file while
> copying, the copy time would decrease.  I'm finding it difficult waiting for
> large data sets to copy over.
>
> I am currently doing:
>
> hadoop dfs -copyFromLocal "/copy/from/path/" input
>
> and am wondering if it's possible to also specify something like -setrep on
> the same line.  -setsrep requires you to specify the file, which implies
> that it has to exist first, requiring two separate commands.
>
> Thanks in advance,
> -Julian

--
Harsh J
+
Julian Bui 2013-05-31, 19:27