Hello,

So I have a question about changing dfs.block.size in
$HADOOP_HOME/conf/hdfs-site.xml.  I understand that when files are created,
blocksizes can be modified from default.  What happens if you modify the
blocksize of an existing HDFS site?  Do newly created files get the default
blocksize and old files remain the same?  Is there a way to change the
blocksize of existing files; I'm assuming you could write MapReduce job to
do it, but any build in facilities?

Thanks,
-JR

hadoop fs -setrep

Sent from my iPhone

On Jun 6, 2011, at 12:09 PM, "J. Ryan Earl" <[EMAIL PROTECTED]> wrote:

> Hello,
>
> So I have a question about changing dfs.block.size in $HADOOP_HOME/conf/hdfs-site.xml.  I understand that when files are created, blocksizes can be modified from default.  What happens if you modify the blocksize of an existing HDFS site?  Do newly created files get the default blocksize and old files remain the same?  Is there a way to change the blocksize of existing files; I'm assuming you could write MapReduce job to do it, but any build in facilities?
>
> Thanks,
> -JR
>
>

Sorry, that's rep factor and not blocksize. I think you need to copy the files.

Sent from my iPhone

On Jun 6, 2011, at 12:09 PM, "J. Ryan Earl" <[EMAIL PROTECTED]> wrote:

> Hello,
>
> So I have a question about changing dfs.block.size in $HADOOP_HOME/conf/hdfs-site.xml.  I understand that when files are created, blocksizes can be modified from default.  What happens if you modify the blocksize of an existing HDFS site?  Do newly created files get the default blocksize and old files remain the same?  Is there a way to change the blocksize of existing files; I'm assuming you could write MapReduce job to do it, but any build in facilities?
>
> Thanks,
> -JR
>
>

I think that you run several maintenance tasks after doing these changes.
* Start the balancer tool to redistribute the blocks by moving them from
over-utilized datanodes to under-utilized datanodes.
    Rebember to change the dfs.balance.bandwidthPerSec property in the
hdfs-site.xml file.

* Run the fsck tool (everyday if it is possible) to check the health of
files in HDFS.

Regards

El 6/6/2011 3:29 PM, Jeff Bean escribi�:
> Sorry, that's rep factor and not blocksize. I think you need to copy the files.
>
> Sent from my iPhone
>
> On Jun 6, 2011, at 12:09 PM, "J. Ryan Earl"<[EMAIL PROTECTED]>  wrote:
>
>    
>> Hello,
>>
>> So I have a question about changing dfs.block.size in $HADOOP_HOME/conf/hdfs-site.xml.  I understand that when files are created, blocksizes can be modified from default.  What happens if you modify the blocksize of an existing HDFS site?  Do newly created files get the default blocksize and old files remain the same?  Is there a way to change the blocksize of existing files; I'm assuming you could write MapReduce job to do it, but any build in facilities?
>>
>> Thanks,
>> -JR
>>
>>
>>      

--
Marcos Lu�s Ort�z Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186

Another advice here, is that you can test the right block size with a
seemed enviroment to your production system, before to deploy the real
system, and then, you can avoid these kinds of changes.

El 6/6/2011 3:09 PM, J. Ryan Earl escribi�:
> Hello,
>
> So I have a question about changing dfs.block.size in
> $HADOOP_HOME/conf/hdfs-site.xml.  I understand that when files are
> created, blocksizes can be modified from default.  What happens if you
> modify the blocksize of an existing HDFS site?  Do newly created files
> get the default blocksize and old files remain the same?  Is there a
> way to change the blocksize of existing files; I'm assuming you could
> write MapReduce job to do it, but any build in facilities?
>
> Thanks,
> -JR
>
>

--
Marcos Lu�s Ort�z Valmaseda
  Software Engineer (UCI)
  http://marcosluis2186.posterous.com
  http://twitter.com/marcosluis2186

On Mon, Jun 6, 2011 at 2:56 PM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:

> Another advice here, is that you can test the right block size with a
> seemed enviroment to your production system, before to deploy the real
> system, and then, you can avoid these kinds of changes.
>

What do you mean by "seemed" environment?  I have mockup/test environments
if that's what you mean.

Thanks,
-JR

 Do newly created files get the default blocksize and old files remain the same? Yes

Is there a way to change the blocksize of existing files? I have done this using copy-out and copy back in script. Couldn't find a short-cut analogous to setrep.

 
-Ayon
See My Photos on Flickr
Also check out my Blog for answers to commonly asked questions.

________________________________
From: J. Ryan Earl <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Monday, June 6, 2011 12:09 PM
Subject: Changing dfs.block.size
Hello,

So I have a question about changing dfs.block.size in $HADOOP_HOME/conf/hdfs-site.xml.  I understand that when files are created, blocksizes can be modified from default.  What happens if you modify the blocksize of an existing HDFS site?  Do newly created files get the default blocksize and old files remain the same?  Is there a way to change the blocksize of existing files; I'm assuming you could write MapReduce job to do it, but any build in facilities?

Thanks,
-JR

On Mon, Jun 6, 2011 at 3:08 PM, Ayon Sinha <[EMAIL PROTECTED]> wrote:

>  Do newly created files get the default blocksize and old files remain the
> same? Yes
> Is there a way to change the blocksize of existing files? I have done this
> using copy-out and copy back in script. Couldn't find a short-cut analogous
> to setrep.
>

Thanks for the information, that was inline with my expectation,
-JR

On Jun 6, 2011, at 12:09 PM, J. Ryan Earl wrote:

> Hello,
>
> So I have a question about changing dfs.block.size in
> $HADOOP_HOME/conf/hdfs-site.xml.  I understand that when files are created,
> blocksizes can be modified from default.  What happens if you modify the
> blocksize of an existing HDFS site?  Do newly created files get the default
> blocksize and old files remain the same?

Yes.
>  Is there a way to change the
> blocksize of existing files; I'm assuming you could write MapReduce job to
> do it, but any build in facilities?

You can use distcp to copy the files back onto the same fs in a new location.  The new files should be in the new block size.  Now you can move the new files where the old files used to live.