Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?


Copy link to this message
-
Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?
Thanks very much, I suppose I know what I should do with
On Mon, Dec 16, 2013 at 5:27 PM, Vinayakumar B <[EMAIL PROTECTED]>wrote:

>  Hi,
>
>
>
> *hadoop.tmp.dir* is not the exact configuration you are looking for
> spreading the disk I/O
>
>
>
> This is the default base directory ( its single directory not multiple)
> used in case you didn’t configure your own directories for processes such
> as NameNode, DataNode and NodeManager.
>
>
>
> Exact configurations where you need to configure comma separated values
> are as follows.
>
>  *1.       **dfs.namenode.name.dir* for  namenode in *hdfs-site.xml*
>
> *2.       **dfs.datanode.data.dir *for datanode in *hdfs-site.xml*
>
> *3.       **yarn.nodemanager.local-dirs* for NodeManager in
> *yarn-site.xml*
>
>
>
> Please note all above configurations are for Hadoop 2.x
>
>
>
> Configure different subdirectories if you are using same disk for multiple
> processes.
>
>                 Ex: /hadoop/data1/dfs/data
>
>                         And
>
>                      /hadoop/data1/yarn/nm-local-dir
>
>
>
>
>
> Cheers,
>
> Vinayakumar B
>
> *From:* Tao Xiao [mailto:[EMAIL PROTECTED]]
> *Sent:* 16 December 2013 14:42
> *To:* [EMAIL PROTECTED]
> *Subject:* Re: How to set "hadoop.tmp.dir" if I have multiple disks per
> node?
>
>
>
> Thanks.
>
> In order to spread I/O among multiple disks, should I assign a
> comma-separated list of directories which are located on different disks to
> "hadoop.tmp.dir"?
>
> for example,
>
>  <property>
>
>       <name>hadoop.tmp.dir</name>
>
>
> <value>/mnt/disk1/hadoop_tmp_dir,/mnt/disk2/hadoop_tmp_dir,/mnt/disk3/hadoop_tmp_dir</value>
>
>  </property>
>
>
>
> 2013/12/16 Shekhar Sharma <[EMAIL PROTECTED]>
>
> hadoop.tmp.dir is a directory created on local file system
> For example if you have set hadoop.tmp.dir property to
> /home/training/hadoop
>
> This directory will be created when you format the namenode by running
> the command
> hadoop namenode -format
>
> When you open this folder
>
>
> you will see two subfolders dfs and mapred.
>
> the /home/training/hadoop/mapred folder will be on HDFS also
>
> Hope this clears
> Regards,
> Som Shekhar Sharma
> +91-8197243810
>
>
>
> On Mon, Dec 16, 2013 at 1:42 PM, Dieter De Witte <[EMAIL PROTECTED]>
> wrote:
> > Hi,
> >
> > Make sure to also set mapred.local.dir to the same set of output
> > directories, this is were the intermediate key-value pairs are stored!
> >
> > Regards, Dieter
> >
> >
> > 2013/12/16 Tao Xiao <[EMAIL PROTECTED]>
> >>
> >> I have ten disks per node,and I don't know what value I should set to
> >> "hadoop.tmp.dir". Some said this property refers to a location in local
> disk
> >> while some other said it refers to a directory in HDFS. I'm confused,
> who
> >> can explain it ?
> >>
> >> I want to spread I/O since I have ten disks per node, so should I set a
> >> comma-separated list of directories (which are on different disks) to
> >> "hadoop.tmp.dir" ?
> >
> >
>
>
>