Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> odd temp dir created in hdfs from hdfs-site.xml file setting?


Copy link to this message
-
Re: odd temp dir created in hdfs from hdfs-site.xml file setting?
Hadoop's default configurations use hadoop.tmp.dir both for the local file
system and HDFS.  Specifically, mapred.system.dir is, I believe, a directory
on HDFS, and defaults to be a subdirectory of hadoop.tmp.dir.

-- Philip

[130]air::~(16603)$grep -C 3 hadoop.tmp.dir
/home/philip/hadoop/hadoop-0.20.2+737/src/mapred/mapred-default.xml

<property>
  <name>mapred.local.dir</name>
  <value>${hadoop.tmp.dir}/mapred/local</value>
  <description>The local directory where MapReduce stores intermediate
  data files.  May be a comma-separated list of
  directories on different devices in order to spread disk i/o.
--

<property>
  <name>mapred.system.dir</name>
  <value>${hadoop.tmp.dir}/mapred/system</value>
  <description>The directory where MapReduce stores control files.
  </description>
</property>

<property>
  <name>mapreduce.jobtracker.staging.root.dir</name>
  <value>${hadoop.tmp.dir}/mapred/staging</value>
  <description>The root of the staging area for users' job files
  In practice, this should be the directory where users' home
  directories are located (usually /user)
--

<property>
  <name>mapred.temp.dir</name>
  <value>${hadoop.tmp.dir}/mapred/temp</value>
  <description>A shared directory for temporary files.
  </description>
</property>
On Wed, Dec 8, 2010 at 5:03 PM, Aaron Eng <[EMAIL PROTECTED]> wrote:

> hadoop.tmp.dir is supposed a location the local file system, not a
> directory within HDFS.
>
> That value should also be in core-site.xml instead of hdfs-site.xml.  That
> might be the cause of your issue.
>
>
> On Wed, Dec 8, 2010 at 4:47 PM, Hiller, Dean (Contractor) <
> [EMAIL PROTECTED]> wrote:
>
>> I have this setting in hdfs-site.xml
>>
>>
>>
>> <property>
>>
>>   <name>hadoop.tmp.dir</name>
>>
>>   <value>/opt/data/hadooptmp</value>
>>
>> </property>
>>
>>
>>
>> This then causes a creation of that dir in HDFS(ie. I can see it when
>> browsing
>>
>>
>>
>> http://206.88.43.4:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=/
>>
>>
>>
>> Is it me, or is that weird/odd?  Do I have some kind of other setting
>> wrong?  Is hadoop.tmp.dir not for local file system like I thought it
>> was(ie. The next property I have after that is stored on the local
>> filesystem..
>>
>>
>>
>> <property>
>>
>>   <name>dfs.data.dir</name>
>>
>>   <value>/opt/data/hadoop</value>
>>
>> </property>
>>
>>
>>
>> Thanks,
>>
>> Dean
>>
>> This message and any attachments are intended only for the use of the addressee and
>> may contain information that is privileged and confidential. If the reader of the
>> message is not the intended recipient or an authorized representative of the
>> intended recipient, you are hereby notified that any dissemination of this
>> communication is strictly prohibited. If you have received this communication in
>> error, please notify us immediately by e-mail and delete the message and any
>> attachments from your system.
>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB