As I read in Hadoop tutorial, Hadoop replicate file blocks by a factor (default 3), in other words, it replicate each block file 3 times. Does Hadoop do it for all files? I mean files written by reducers are replicated too?

The replication factor is per-file (and not HDFS-wide), and can be
controlled by setting "dfs.replication" in your job config to the
desired amount, to affect the whole job.

If you want to use config files itself to propagate this, place your
chosen "dfs.replication" default value inside conf/hdfs-site.xml.

On Tue, Jan 31, 2012 at 5:49 PM, Alieh Saeedi <[EMAIL PROTECTED]> wrote:
> As I read in Hadoop tutorial, Hadoop replicate file blocks by a factor
> (default 3), in other words, it replicate each block file 3 times. Does
> Hadoop do it for all files? I mean files written by reducers are replicated
> too?

Yes, all files written to HDFS will be replicated, but you can control
your # of replicas as detailed on top of post. Setting replication
factor to 1 would mean no replication.

--
Harsh J
Customer Ops. Engineer, Cloudera

Hi,

 

I have set replication as 1 in hdfs-site.xml as given below,

 

property>

    <name>dfs.replication</name>

    <value>1</value>

    <description>Default block replication.

       The actual number of replications can be specified when the file is
created.

       The default is used if replication is not specified in create time.

    </description>

 </property>

 

But I get the below errors when I retrieve the block report from hdfs. Can
someone please help me if I am missing anything.

 

"Under replicated blk_4884628009930930282_210741. Target Replicas is 3 but
found 2 replica(s)."

 

Thanks

Ajay

 

Ajay,

Replication is a per-file property.

To lower replication factor of existing files, run (as an HDFS
superuser, for a good measure):
$ hadoop fs -setrep -R 3 /

On Wed, Feb 15, 2012 at 5:39 PM, ajay.bhosle <[EMAIL PROTECTED]> wrote:
> Hi,
>
>
>
> I have set replication as 1 in hdfs-site.xml as given below,
>
>
>
> property>
>
>     <name>dfs.replication</name>
>
>     <value>1</value>
>
>     <description>Default block replication.
>
>        The actual number of replications can be specified when the file is
> created.
>
>        The default is used if replication is not specified in create time.
>
>     </description>
>
>  </property>
>
>
>
> But I get the below errors when I retrieve the block report from hdfs. Can
> someone please help me if I am missing anything.
>
>
>
> “Under replicated blk_4884628009930930282_210741. Target Replicas is 3 but
> found 2 replica(s).”
>
>
>
> Thanks
>
> Ajay
>
>

--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

hi Ajay
    Does the file that related with that
blockId(blk_4884628009930930282_210741) has being existed at HDFS?
    Your setting is right for new file to HDFS after the configuration to
take effect.

-Regards
Denny Ye

2012/2/15 Harsh J <[EMAIL PROTECTED]>

> Ajay,
>
> Replication is a per-file property.
>
> To lower replication factor of existing files, run (as an HDFS
> superuser, for a good measure):
> $ hadoop fs -setrep -R 3 /
>
> On Wed, Feb 15, 2012 at 5:39 PM, ajay.bhosle <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> >
> >
> > I have set replication as 1 in hdfs-site.xml as given below,
> >
> >
> >
> > property>
> >
> >     <name>dfs.replication</name>
> >
> >     <value>1</value>
> >
> >     <description>Default block replication.
> >
> >        The actual number of replications can be specified when the file
> is
> > created.
> >
> >        The default is used if replication is not specified in create
> time.
> >
> >     </description>
> >
> >  </property>
> >
> >
> >
> > But I get the below errors when I retrieve the block report from hdfs.
> Can
> > someone please help me if I am missing anything.
> >
> >
> >
> > “Under replicated blk_4884628009930930282_210741. Target Replicas is 3
> but
> > found 2 replica(s).”
> >
> >
> >
> > Thanks
> >
> > Ajay
> >
> >
>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
>

Hi Denny/Harsh,

 

Apart from the below configuration mentioned by me, I also ran the setrep
command to set the replication for existing files. But after some days when
I get the block location report it again has a long list of errors
mentioning "Target Replicas is 3 but > found 2 replica(s)." .

 

Thanks

Ajay

 

From: Denny Ye [mailto:[EMAIL PROTECTED]]
Sent: Friday, February 17, 2012 8:10 AM
To: [EMAIL PROTECTED]
Subject: Re: Replication

 

hi Ajay

    Does the file that related with that
blockId(blk_4884628009930930282_210741) has being existed at HDFS?

    Your setting is right for new file to HDFS after the configuration to
take effect.

 

-Regards

Denny Ye

 

2012/2/15 Harsh J <[EMAIL PROTECTED]>

Ajay,

Replication is a per-file property.

To lower replication factor of existing files, run (as an HDFS
superuser, for a good measure):
$ hadoop fs -setrep -R 3 /
On Wed, Feb 15, 2012 at 5:39 PM, ajay.bhosle <[EMAIL PROTECTED]>
wrote:
> Hi,
>
>
>
> I have set replication as 1 in hdfs-site.xml as given below,
>
>
>
> property>
>
>     <name>dfs.replication</name>
>
>     <value>1</value>
>
>     <description>Default block replication.
>
>        The actual number of replications can be specified when the file is
> created.
>
>        The default is used if replication is not specified in create time.
>
>     </description>
>
>  </property>
>
>
>
> But I get the below errors when I retrieve the block report from hdfs. Can
> someone please help me if I am missing anything.
>
>
>
> "Under replicated blk_4884628009930930282_210741. Target Replicas is 3 but
> found 2 replica(s)."
>
>
>
> Thanks
>
> Ajay
>
>
--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

 

Ajay,

Some of your client programs are creating files with replication
factor 3 explicitly (or are using a default configuration instead of
your configuration, which sets it to 3). The dfs.replication factor
config isn't cluster-wide as I mentioned before, and a client may
override it (or override via a direct integer value via the Java API)
at will. You need to fix this from the client side.

On Fri, Feb 17, 2012 at 11:42 AM, ajay.bhosle <[EMAIL PROTECTED]> wrote:
> Hi Denny/Harsh,
>
>
>
> Apart from the below configuration mentioned by me, I also ran the setrep
> command to set the replication for existing files. But after some days when
> I get the block location report it again has a long list of errors
> mentioning “Target Replicas is 3 but > found 2 replica(s).” .
>
>
>
> Thanks
>
> Ajay
>
>
>
> From: Denny Ye [mailto:[EMAIL PROTECTED]]
> Sent: Friday, February 17, 2012 8:10 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Replication
>
>
>
> hi Ajay
>
>     Does the file that related with that
> blockId(blk_4884628009930930282_210741) has being existed at HDFS?
>
>     Your setting is right for new file to HDFS after the configuration to
> take effect.
>
>
>
> -Regards
>
> Denny Ye
>
>
>
> 2012/2/15 Harsh J <[EMAIL PROTECTED]>
>
> Ajay,
>
> Replication is a per-file property.
>
> To lower replication factor of existing files, run (as an HDFS
> superuser, for a good measure):
> $ hadoop fs -setrep -R 3 /
>
>
> On Wed, Feb 15, 2012 at 5:39 PM, ajay.bhosle <[EMAIL PROTECTED]>
> wrote:
>> Hi,
>>
>>
>>
>> I have set replication as 1 in hdfs-site.xml as given below,
>>
>>
>>
>> property>
>>
>>     <name>dfs.replication</name>
>>
>>     <value>1</value>
>>
>>     <description>Default block replication.
>>
>>        The actual number of replications can be specified when the file is
>> created.
>>
>>        The default is used if replication is not specified in create time.
>>
>>     </description>
>>
>>  </property>
>>
>>
>>
>> But I get the below errors when I retrieve the block report from hdfs. Can
>> someone please help me if I am missing anything.
>>
>>
>>
>> “Under replicated blk_4884628009930930282_210741. Target Replicas is 3 but
>> found 2 replica(s).”
>>
>>
>>
>> Thanks
>>
>> Ajay
>>
>>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
>
>

--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about

Harsh,

That was the issue. Thanks a lot.

Ajay

-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Friday, February 17, 2012 12:11 PM
To: [EMAIL PROTECTED]
Subject: Re: Replication

Ajay,

Some of your client programs are creating files with replication
factor 3 explicitly (or are using a default configuration instead of
your configuration, which sets it to 3). The dfs.replication factor
config isn't cluster-wide as I mentioned before, and a client may
override it (or override via a direct integer value via the Java API)
at will. You need to fix this from the client side.

On Fri, Feb 17, 2012 at 11:42 AM, ajay.bhosle <[EMAIL PROTECTED]>
wrote:
> Hi Denny/Harsh,
>
>
>
> Apart from the below configuration mentioned by me, I also ran the setrep
> command to set the replication for existing files. But after some days
when
> I get the block location report it again has a long list of errors
> mentioning “Target Replicas is 3 but > found 2 replica(s).” .
>
>
>
> Thanks
>
> Ajay
>
>
>
> From: Denny Ye [mailto:[EMAIL PROTECTED]]
> Sent: Friday, February 17, 2012 8:10 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Replication
>
>
>
> hi Ajay
>
>     Does the file that related with that
> blockId(blk_4884628009930930282_210741) has being existed at HDFS?
>
>     Your setting is right for new file to HDFS after the configuration to
> take effect.
>
>
>
> -Regards
>
> Denny Ye
>
>
>
> 2012/2/15 Harsh J <[EMAIL PROTECTED]>
>
> Ajay,
>
> Replication is a per-file property.
>
> To lower replication factor of existing files, run (as an HDFS
> superuser, for a good measure):
> $ hadoop fs -setrep -R 3 /
>
>
> On Wed, Feb 15, 2012 at 5:39 PM, ajay.bhosle <[EMAIL PROTECTED]>
> wrote:
>> Hi,
>>
>>
>>
>> I have set replication as 1 in hdfs-site.xml as given below,
>>
>>
>>
>> property>
>>
>>     <name>dfs.replication</name>
>>
>>     <value>1</value>
>>
>>     <description>Default block replication.
>>
>>        The actual number of replications can be specified when the file
is
>> created.
>>
>>        The default is used if replication is not specified in create
time.
>>
>>     </description>
>>
>>  </property>
>>
>>
>>
>> But I get the below errors when I retrieve the block report from hdfs.
Can
>> someone please help me if I am missing anything.
>>
>>
>>
>> “Under replicated blk_4884628009930930282_210741. Target Replicas is 3
but
>> found 2 replica(s).”
>>
>>
>>
>> Thanks
>>
>> Ajay
>>
>>
>
>
> --
> Harsh J
> Customer Ops. Engineer
> Cloudera | http://tiny.cloudera.com/about
>
>

--
Harsh J
Customer Ops. Engineer
Cloudera | http://tiny.cloudera.com/about