-Re: Creating and working with temporary file in a map() function
Harsh J 2012-04-07, 21:29
MapReduce sets "mapred.child.tmp" for all tasks to be the Task
Attempt's WorkingDir/tmp automatically. This also sets the
-Djava.io.tmpdir prop for each task at JVM boot.
Hence you may use the regular Java API to create a temporary file:
These files would also be automatically deleted away after the task
attempt is done.
On Sun, Apr 8, 2012 at 2:14 AM, Ondřej Klimpera <[EMAIL PROTECTED]> wrote:
> I would like to ask you if it is possible to create and work with a
> temporary file while in a map function.
> I suppose that map function is running on a single node in Hadoop cluster.
> So what is a safe way to create a temporary file and read from it in one
> map() run. If it is possible is there a size limit for the file.
> The file can not be created before hadoop job is created. I need to create
> and process the file inside map().
> Thanks for your answer.
> Ondrej Klimpera.