Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - How to set "hadoop.tmp.dir" if I have multiple disks per node?


Copy link to this message
-
Re: How to set "hadoop.tmp.dir" if I have multiple disks per node?
Azuryy Yu 2013-12-16, 09:25
Hi Tao,

No, you need to set mapred.local.dir  in the mapred-site.xml with comma
separated list of path to spread I/O .
On Mon, Dec 16, 2013 at 5:11 PM, Tao Xiao <[EMAIL PROTECTED]> wrote:

> Thanks.
>
> In order to spread I/O among multiple disks, should I assign a
> comma-separated list of directories which are located on different disks to
> "hadoop.tmp.dir"?
>
> for example,
>
>  <property>
>       <name>hadoop.tmp.dir</name>
>
> <value>/mnt/disk1/hadoop_tmp_dir,/mnt/disk2/hadoop_tmp_dir,/mnt/disk3/hadoop_tmp_dir</value>
>  </property>
>
>
> 2013/12/16 Shekhar Sharma <[EMAIL PROTECTED]>
>
>> hadoop.tmp.dir is a directory created on local file system
>> For example if you have set hadoop.tmp.dir property to
>> /home/training/hadoop
>>
>> This directory will be created when you format the namenode by running
>> the command
>> hadoop namenode -format
>>
>> When you open this folder
>>
>>
>> you will see two subfolders dfs and mapred.
>>
>> the /home/training/hadoop/mapred folder will be on HDFS also
>>
>> Hope this clears
>> Regards,
>> Som Shekhar Sharma
>> +91-8197243810
>>
>>
>> On Mon, Dec 16, 2013 at 1:42 PM, Dieter De Witte <[EMAIL PROTECTED]>
>> wrote:
>> > Hi,
>> >
>> > Make sure to also set mapred.local.dir to the same set of output
>> > directories, this is were the intermediate key-value pairs are stored!
>> >
>> > Regards, Dieter
>> >
>> >
>> > 2013/12/16 Tao Xiao <[EMAIL PROTECTED]>
>> >>
>> >> I have ten disks per node,and I don't know what value I should set to
>> >> "hadoop.tmp.dir". Some said this property refers to a location in
>> local disk
>> >> while some other said it refers to a directory in HDFS. I'm confused,
>> who
>> >> can explain it ?
>> >>
>> >> I want to spread I/O since I have ten disks per node, so should I set a
>> >> comma-separated list of directories (which are on different disks) to
>> >> "hadoop.tmp.dir" ?
>> >
>> >
>>
>
>