Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: temporary folders for YARN tasks


Copy link to this message
-
Re: temporary folders for YARN tasks
Sandy Ryza 2013-07-02, 06:12
LocalDirAllocator should help with this.  You can look through MapReduce
code to see how it's used.

-Sandy
On Mon, Jul 1, 2013 at 11:01 PM, Devaraj k <[EMAIL PROTECTED]> wrote:

>  You can make use of this configuration to do the same. ****
>
> ** **
>
> <property>****
>
>     <description>List of directories to store *localized* files in. An ***
> *
>
>       application's *localized* file directory will be found in:****
>
>       ${yarn.nodemanager.local-*dirs*}/*usercache*/${user}/*appcache*
> /application_${*appid*}.****
>
>       Individual containers' work directories, called container_${*contid*},
> will****
>
>       be *subdirectories* of this.****
>
>    </description>****
>
>     <name>yarn.nodemanager.local-*dirs*</name>****
>
>     <value>${hadoop.tmp.dir}/*nm*-local-*dir*</value>****
>
>   </property>****
>
> ** **
>
> Thanks****
>
> Devaraj k****
>
> ** **
>
> *From:* John Lilley [mailto:[EMAIL PROTECTED]]
> *Sent:* 02 July 2013 02:08
> *To:* [EMAIL PROTECTED]
> *Subject:* temporary folders for YARN tasks****
>
> ** **
>
> When a YARN app and its tasks wants to write temporary files, how does it
> know where to write the files?  ****
>
> I am assuming that each task has some temporary space available, and I
> hope it is available across multiple disk volumes for parallel performance.
> ****
>
> Are those files cleaned up automatically after task exit?****
>
> If I want to give lifetime control of the files to an auxiliary service
> (along the lines of MR shuffle passing files to the aux service), how would
> I do that, and would that entail different file locations?****
>
> Thanks****
>
> John****
>
> ** **
>
> ** **
>