Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: final the dfs.replication and fsck


Copy link to this message
-
Re: final the dfs.replication and fsck
Thanks Harsh, dfs.replication.max does do the magic!!

On Mon, Oct 15, 2012 at 1:19 PM, Chris Nauroth <[EMAIL PROTECTED]> wrote:
> Thank you, Harsh.  I did not know about dfs.replication.max.
>
>
> On Mon, Oct 15, 2012 at 12:23 PM, Harsh J <[EMAIL PROTECTED]> wrote:
>>
>> Hey Chris,
>>
>> The dfs.replication param is an exception to the <final> config
>> feature. If one uses the FileSystem API, one can pass in any short
>> value they want the replication to be. This bypasses the
>> configuration, and the configuration (being per-file) is also client
>> sided.
>>
>> The right way for an administrator to enforce a "max" replication
>> value at a create/setRep level, would be to set
>> the dfs.replication.max to a desired value at the NameNode and restart
>> it.
>>
>> On Tue, Oct 16, 2012 at 12:48 AM, Chris Nauroth
>> <[EMAIL PROTECTED]> wrote:
>> > Hello Patai,
>> >
>> > Has your configuration file change been copied to all nodes in the
>> > cluster?
>> >
>> > Are there applications connecting from outside of the cluster?  If so,
>> > then
>> > those clients could have separate configuration files or code setting
>> > dfs.replication (and other configuration properties).  These would not
>> > be
>> > limited by final declarations in the cluster's configuration files.
>> > <final>true</final> controls configuration file resource loading, but it
>> > does not necessarily block different nodes or different applications
>> > from
>> > running with completely different configurations.
>> >
>> > Hope this helps,
>> > --Chris
>> >
>> >
>> > On Mon, Oct 15, 2012 at 12:01 PM, Patai Sangbutsarakum
>> > <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Hi Hadoopers,
>> >>
>> >> I have
>> >> <property>
>> >>     <name>dfs.replication</name>
>> >>     <value>2</value>
>> >>     <final>true</final>
>> >>   </property>
>> >>
>> >> set in hdfs-site.xml in staging environment cluster. while the staging
>> >> cluster is running the code that will later be deployed in production,
>> >> those code is trying to have dfs.replication of 3, 10, 50, other than
>> >> 2; the number that developer thought that will fit in production
>> >> environment.
>> >>
>> >> Even though I final the property dfs.replication in staging cluster
>> >> already. every time i run fsck on the staging cluster i still see it
>> >> said under replication.
>> >> I thought final keyword will not honor value in job config, but it
>> >> doesn't seem so when i run fsck.
>> >>
>> >> I am on cdh3u4.
>> >>
>> >> please suggest.
>> >> Patai
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB