-Re: can the parameters dfs.block.size and dfs.replication be different from one file to the other
Shahab Yunus 2013-09-10, 18:38
"can be set at the time I load the file to the HDFS (that is, it is the
client side setting)? "
I don't think you can do this while reading. These are done at the time of
You can do it like this (the example is for CLI as evident):
hadoop fs -D fs.local.block.size=134217728 -put local_name remote_location
Same is applicable with replication property.
So given that, you I think you have to modify the FileOutputFormat (and
other 'writing' classes) to allow these to be configurable at the time
files are being generated by M/R
On Tue, Sep 10, 2013 at 2:08 PM, Jun Li <[EMAIL PROTECTED]> wrote:
> I am trying to evaluate the MapReduce with different setting. I wonder
> whether the following two HDFS parameters:
> can be set at the time I load the file to the HDFS (that is, it is the
> client side setting)? or these are the system parameter settings that can
> not be changed from the HDFS client invocation.
> I am using Hadoop 1.1.2 (the recent stable release), rather than the new
> Hadoop 2.x. By reading the Cloudera document, I wonder even if such
> parameters can be set per HDFS client, will it be supported only after
> certain Hadoop version?
> Thank you!