Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Are hadoop fs commands serial or parallel

Copy link to this message
Re: Are hadoop fs commands serial or parallel
The sequence file writer definitely does it serially as you can only
ever write to the end of a file in Hadoop.

Doing copyFromLocal could write multiple files in parallel (I'm not
sure if it does or not), but a single file would be written serially.


On Tue, May 17, 2011 at 5:44 PM, Mapred Learn <[EMAIL PROTECTED]> wrote:
>  Hi,
> My question is when I run a command from hdfs client, for eg. hadoop fs
> -copyFromLocal or create a sequence file writer in java code and append
> key/values to it through Hadoop APIs, does it internally transfer/write data
> to HDFS serially or in parallel ?
> Thanks in advance,
> -JJ

Joseph Echeverria
Cloudera, Inc.