Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - "JVM reuse": if I use this option, do setup(), cleanup() get called only once?


Copy link to this message
-
Re: "JVM reuse": if I use this option, do setup(), cleanup() get called only once?
Harsh J 2012-11-13, 08:20
Hi,

Those API hooks are called once per task attempt, and regardless of
JVM reuse they will still be run once per task attempt. So yes,
setup+cleanup for every map split or reduce partition that runs
through the reused JVM.

On Tue, Nov 13, 2012 at 1:47 PM, edward choi <[EMAIL PROTECTED]> wrote:
> Hi,
>
> This question might sound stupid, but I couldn't find a definite answer on
> Google.
> My job loads a big file at setup() in Map tasks.
> So I would like to use the loaded file again and again.
>
> I came across "JVM reuse", but text book only says that this option enables
> multiple use of
> tasks on a single JVM. It does not say anything about setup() or cleanup().
>
> Even if I set "mapred.job.reuse.jvm.num.tasks" to -1, do setup() and
> cleanup() get called every single time a task is launched?
>
> Best,
> Ed

--
Harsh J