Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> HDFS to a different location other than HADOOP HOME


+
Raj Hadoop 2013-06-11, 14:22
+
Mohammad Mustaqeem 2013-06-11, 14:28
+
Raj Hadoop 2013-06-11, 14:42
+
Mohammad Tariq 2013-06-11, 16:49
+
Shahab Yunus 2013-06-11, 17:33
+
Mohammad Tariq 2013-06-11, 17:44
+
Sandeep L 2013-06-12, 05:48
+
Sandeep L 2013-06-12, 05:55
+
Pramod N 2013-06-11, 17:19
Copy link to this message
-
Re: HDFS to a different location other than HADOOP HOME
 
But Tariq - /SD1/hadoop_data is in a separate group.
 
My hadoop is in group 'grp1'
Filesystem /SD1 is under group 'grp2'
 
As I have space under SD1, I want to use SD1 for hadoop.
 
so setting permissions like '755' on /SD1/hadoop_data -> will it work?
________________________________
From: Mohammad Tariq <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Raj Hadoop <[EMAIL PROTECTED]>
Sent: Friday, June 14, 2013 1:44 PM
Subject: Re: HDFS to a different location other than HADOOP HOME

Change the permissions of /SD1/hadoop_data to 755 and restart the process.
Warm Regards,
Tariq
cloudfront.blogspot.com

On Fri, Jun 14, 2013 at 11:10 PM, Raj Hadoop <[EMAIL PROTECTED]> wrote:
I have modified dfs.data.dir from the default value to another value which is outside 'HADOOP_HOME' and it is >/SD1/hadoop_data.

><property>
><name>dfs.data.dir</name>
><value>/SD1/hadoop_data</value>
><description>Where DataNodes store their blocks</description>
></property>

>My data node is not starting after the above change. Can anyone tell whats the issue?
>
>
>From: Sandeep L <[EMAIL PROTECTED]>
>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>Sent: Wednesday, June 12, 2013 1:48 AM
>Subject: RE: HDFS to a different location other than HADOOP HOME
>
>Instead of using ${hadoop.tmp.dir}/dfs/data you give multiple directories with absolute paths.
>Just try as follows:
><property>
><name>hadoop.tmp.dir</name>
><value>/home1/user1/dir1,/home2/user2/dir2</value>
></property>
>This will allow HDFS to write data into all specified directories.
>From: [EMAIL PROTECTED]ate: Tue, 11 Jun 2013 23:14:08 +0530Subject: Re: HDFS to a different location other than HADOOP HOMETo: [EMAIL PROTECTED]
>Thank you Pramod and Shahab.
>Warm Regards,
>Tariq
>cloudfront.blogspot.com
>
>On Tue, Jun 11, 2013 at 11:03 PM, Shahab Yunus <[EMAIL PROTECTED]> wrote:
>http://hadoop.apache.org/docs/stable/hdfs-default.html
>>dfs.data.dir ${hadoop.tmp.dir}/dfs/data Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named directories, typically on different devices. Directories that do not exist are ignored.
>>On Tue, Jun 11, 2013 at 1:08 PM, Raj Hadoop <[EMAIL PROTECTED]> wrote:
>>Hi Tariq,
>>> 
>>>What is the default value of dfs.data.dir? My hdfs-site.xml doesnt have this value defined. So what is the default value?
>>> 
>>>Thanks,
>>>Raj
>>>From: Mohammad Tariq <[EMAIL PROTECTED]>
>>>To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Raj Hadoop <[EMAIL PROTECTED]>
>>>Sent: Tuesday, June 11, 2013 12:49 PM
>>>Subject: Re: HDFS to a different location other than HADOOP HOME
>>>
>>>Hello Raj,
>>>    Although the way which Mustaqeem has told is correct, there is more appropriate way to do this. Add the "dfs.data.dir" property in your hdfs-site.xml and give the "path you want to use for HDFS data storage" as its value. The directory pointed by hadoop.tmp.dir also contains the metadata and some other temporary files, so it's always better to have a dedicated location to store the actual HDFS data.
>>>Warm Regards,
>>>Tariq
>>>cloudfront.blogspot.com
>>>On Tue, Jun 11, 2013 at 8:12 PM, Raj Hadoop <[EMAIL PROTECTED]> wrote:
>>> 
>>>>Thanks Mustaqeem.
>>>> 
>>>>hadoop.tmp.dir - Can this store HDFS files? I mean - is there any difference in files from the one we create under HADOOP_HOME.
>>>> 
>>>> 
>>>>From: Mohammad Mustaqeem <[EMAIL PROTECTED]>
>>>>To: user <[EMAIL PROTECTED]>; Raj Hadoop <[EMAIL PROTECTED]>
>>>>Sent: Tuesday, June 11, 2013 10:28 AM
>>>>Subject: Re: HDFS to a different location other than HADOOP HOME
>>>>
>>>>You can mentioned the "hadoop.tmp.dir" property in the core-site.xml as -
>>>><property>
>>>><name>hadoop.tmp.dir</name>
>>>><value>/app/software/app1</value>
>>>></property>
>>>>"/app/software/app1" will be used as location for storing the hdfs files.
>>>>On Tue, Jun 11, 2013 at 7:52 PM, Raj Hadoop <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB