Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - odd temp dir created in hdfs from hdfs-site.xml file setting?


+
Hiller, Dean 2010-12-09, 00:47
+
Aaron Eng 2010-12-09, 01:03
Copy link to this message
-
Re: odd temp dir created in hdfs from hdfs-site.xml file setting?
Philip Zeyliger 2010-12-09, 01:24
Hadoop's default configurations use hadoop.tmp.dir both for the local file
system and HDFS.  Specifically, mapred.system.dir is, I believe, a directory
on HDFS, and defaults to be a subdirectory of hadoop.tmp.dir.

-- Philip

[130]air::~(16603)$grep -C 3 hadoop.tmp.dir
/home/philip/hadoop/hadoop-0.20.2+737/src/mapred/mapred-default.xml

<property>
  <name>mapred.local.dir</name>
  <value>${hadoop.tmp.dir}/mapred/local</value>
  <description>The local directory where MapReduce stores intermediate
  data files.  May be a comma-separated list of
  directories on different devices in order to spread disk i/o.
--

<property>
  <name>mapred.system.dir</name>
  <value>${hadoop.tmp.dir}/mapred/system</value>
  <description>The directory where MapReduce stores control files.
  </description>
</property>

<property>
  <name>mapreduce.jobtracker.staging.root.dir</name>
  <value>${hadoop.tmp.dir}/mapred/staging</value>
  <description>The root of the staging area for users' job files
  In practice, this should be the directory where users' home
  directories are located (usually /user)
--

<property>
  <name>mapred.temp.dir</name>
  <value>${hadoop.tmp.dir}/mapred/temp</value>
  <description>A shared directory for temporary files.
  </description>
</property>
On Wed, Dec 8, 2010 at 5:03 PM, Aaron Eng <[EMAIL PROTECTED]> wrote:

> hadoop.tmp.dir is supposed a location the local file system, not a
> directory within HDFS.
>
> That value should also be in core-site.xml instead of hdfs-site.xml.  That
> might be the cause of your issue.
>
>
> On Wed, Dec 8, 2010 at 4:47 PM, Hiller, Dean (Contractor) <
> [EMAIL PROTECTED]> wrote:
>
>> I have this setting in hdfs-site.xml
>>
>>
>>
>> <property>
>>
>>   <name>hadoop.tmp.dir</name>
>>
>>   <value>/opt/data/hadooptmp</value>
>>
>> </property>
>>
>>
>>
>> This then causes a creation of that dir in HDFS(ie. I can see it when
>> browsing
>>
>>
>>
>> http://206.88.43.4:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/
>>
>>
>>
>> Is it me, or is that weird/odd?  Do I have some kind of other setting
>> wrong?  Is hadoop.tmp.dir not for local file system like I thought it
>> was(ie. The next property I have after that is stored on the local
>> filesystem..
>>
>>
>>
>> <property>
>>
>>   <name>dfs.data.dir</name>
>>
>>   <value>/opt/data/hadoop</value>
>>
>> </property>
>>
>>
>>
>> Thanks,
>>
>> Dean
>>
>> This message and any attachments are intended only for the use of the addressee and
>> may contain information that is privileged and confidential. If the reader of the
>> message is not the intended recipient or an authorized representative of the
>> intended recipient, you are hereby notified that any dissemination of this
>> communication is strictly prohibited. If you have received this communication in
>> error, please notify us immediately by e-mail and delete the message and any
>> attachments from your system.
>>
>>
>>
>