Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: about replication


+
Irfan Sayed 2013-08-06, 06:50
+
manish dunani 2013-08-06, 07:10
+
Irfan Sayed 2013-08-05, 06:53
+
Mohammad Tariq 2013-08-05, 09:33
+
Irfan Sayed 2013-08-05, 11:16
+
Mohammad Tariq 2013-08-05, 11:48
+
Irfan Sayed 2013-08-05, 11:59
Copy link to this message
-
Re: about replication
manish dunani 2013-08-05, 12:30
You can not physically access the datanode.You have to understand it to
logically and it really happens.

Type "jps" command to check ur datanode was started or not.

when user stores the file into hdfs ,the request is goes to datanode and
datanode will divide the file into number of blocks.
Each block size is 64mb or 128mb.but,default is 64 mb.

The default replication factor is 1 that was u already set in ur
hdfs-site.xml .If u want to change ur replication factor from 1 to 3 for
particular file or particular directory in hdfs then use the below commands
appropriately.

First load any file to hdfs using command:

bin/hadoop  dfs -copyFromLocal /ur/local/directory/path/to/file
 /ur/hdfs/directory/path.

*commands:*

To set replication of an individual file to 4:

./bin/hadoop dfs -setrep -w 4 /path/to/file

You can also do this recursively. To change replication of entire HDFS to 1:

./bin/hadoop dfs -setrep -R -w 1 /
*
*
 If the replication factor is 3 then data will divided into 3 blocks and
replicated to various data nodes of ur production cluster(multi node).

Here Replication means same file copied three times on data nodes for
handling hardware failure.
On Mon, Aug 5, 2013 at 5:29 PM, Irfan Sayed <[EMAIL PROTECTED]> wrote:

> thanks.
> please refer below:
>
> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $ ./hadoop dfs -ls /wksp
> Found 1 items
> drwxr-xr-x   - Administrator Domain          0 2013-08-05 16:58 /wksp/New
> folder
>
> Administrator@DFS-DC /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $
>
>
> same command if i run on the datanode then it says:
>
> Administrator@DFS-1 /cygdrive/c/hadoop-1.1.2/hadoop-1.1.2/bin
> $ ./hadoop dfs -ls /wksp
> ls: Cannot access /wksp: No such file or directory.
>
> does it mean that replication is not yet started ...?
>
> please suggest
>
> regards
>
>
>
> On Mon, Aug 5, 2013 at 5:18 PM, Mohammad Tariq <[EMAIL PROTECTED]> wrote:
>
>> You cannot physically see the HDFS files and directories through local
>> FS. Either use HDFS shell or HDFS webUI(namenode_machine:50070).
>>
>> Warm Regards,
>> Tariq
>> cloudfront.blogspot.com
>>
>>
>> On Mon, Aug 5, 2013 at 4:46 PM, Irfan Sayed <[EMAIL PROTECTED]> wrote:
>>
>>> thanks mohammad
>>> i ran the below command on NameNode
>>>
>>> $ ./hadoop dfs -mkdir /wksp
>>>
>>> and the "wksp" dir got created in c:\ ( as i have windows environment)
>>>
>>> now , when i log in to one of the DataNode , then i am not able to see
>>> c:\wksp
>>>
>>> any issue ?
>>> please suggest
>>>
>>> regards
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Aug 5, 2013 at 3:03 PM, Mohammad Tariq <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hello Irfan,
>>>>
>>>> You can find all the answers from HDFS architecture guide<http://hadoop.apache.org/docs/stable/hdfs_design.html>.
>>>> See the section Data Organization<http://hadoop.apache.org/docs/stable/hdfs_design.html#Data+Organization>in particular for this question.
>>>>
>>>> Warm Regards,
>>>> Tariq
>>>> cloudfront.blogspot.com
>>>>
>>>>
>>>> On Mon, Aug 5, 2013 at 12:23 PM, Irfan Sayed <[EMAIL PROTECTED]>wrote:
>>>>
>>>>> hi,
>>>>>
>>>>> i have setup the two node apache hadoop cluster on windows environment
>>>>> one is namenode and another is datanode
>>>>>
>>>>> everything is working fine.
>>>>> one thing which i need to know is , how the replication starts
>>>>> if i create a.txt in namenode , how it will be appeared in datanodes
>>>>>
>>>>> please suggest
>>>>>
>>>>> regards
>>>>>
>>>>>
>>>>
>>>
>>
>
--
MANISH DUNANI
-THANX
+91 9426881954,+91 8460656443
[EMAIL PROTECTED]
+
Irfan Sayed 2013-08-06, 04:59
+
manish dunani 2013-08-06, 06:10
+
Mohammad Tariq 2013-08-06, 20:22
+
Mohammad Tariq 2013-08-06, 11:49
+
Irfan Sayed 2013-08-06, 11:56
+
Mohammad Tariq 2013-08-06, 12:00
+
Irfan Sayed 2013-08-20, 09:26
+
Mohammad Tariq 2013-08-20, 09:53
+
Irfan Sayed 2013-08-20, 10:44
+
Mohammad Tariq 2013-08-21, 11:58
+
Irfan Sayed 2013-08-22, 04:22
+
Arpit Agarwal 2013-08-22, 04:29
+
Irfan Sayed 2013-08-22, 05:05
+
Arpit Agarwal 2013-08-22, 05:56
+
Irfan Sayed 2013-08-22, 06:19
+
Irfan Sayed 2013-08-22, 11:31
+
Mohammad Tariq 2013-08-22, 14:26
+
Irfan Sayed 2013-08-23, 04:15
+
Olivier Renault 2013-08-23, 07:27
+
Irfan Sayed 2013-08-23, 09:54
+
Olivier Renault 2013-08-23, 10:40
+
Olivier Renault 2013-09-03, 07:28