Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: Are hadoop fs commands serial or parallel


Copy link to this message
-
Re: Are hadoop fs commands serial or parallel
The sequence file writer definitely does it serially as you can only
ever write to the end of a file in Hadoop.

Doing copyFromLocal could write multiple files in parallel (I'm not
sure if it does or not), but a single file would be written serially.

-Joey

On Tue, May 17, 2011 at 5:44 PM, Mapred Learn <[EMAIL PROTECTED]> wrote:
>  Hi,
> My question is when I run a command from hdfs client, for eg. hadoop fs
> -copyFromLocal or create a sequence file writer in java code and append
> key/values to it through Hadoop APIs, does it internally transfer/write data
> to HDFS serially or in parallel ?
>
> Thanks in advance,
> -JJ
>

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB