Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> "JVM reuse": if I use this option, do setup(), cleanup() get called only once?


+
edward choi 2012-11-13, 08:17
Copy link to this message
-
Re: "JVM reuse": if I use this option, do setup(), cleanup() get called only once?
Hi,

Those API hooks are called once per task attempt, and regardless of
JVM reuse they will still be run once per task attempt. So yes,
setup+cleanup for every map split or reduce partition that runs
through the reused JVM.

On Tue, Nov 13, 2012 at 1:47 PM, edward choi <[EMAIL PROTECTED]> wrote:
> Hi,
>
> This question might sound stupid, but I couldn't find a definite answer on
> Google.
> My job loads a big file at setup() in Map tasks.
> So I would like to use the loaded file again and again.
>
> I came across "JVM reuse", but text book only says that this option enables
> multiple use of
> tasks on a single JVM. It does not say anything about setup() or cleanup().
>
> Even if I set "mapred.job.reuse.jvm.num.tasks" to -1, do setup() and
> cleanup() get called every single time a task is launched?
>
> Best,
> Ed

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB