Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # general >> Lazy initialization of Reducers


+
Syed Wasti 2010-07-21, 17:15
Copy link to this message
-
Re: Lazy initialization of Reducers
I don't find such parameter in 0.20.2

Please create such flag in your own class.

On Wed, Jul 21, 2010 at 10:15 AM, Syed Wasti <[EMAIL PROTECTED]> wrote:

>
> Hi,
>
> I read about this Reducer Lazy initialization in a document found in the
> below URL.
>
> http://www.scribd.com/doc/23046928/Hadoop-Performance-Tuning
>
>
>
> It says “:In M/R job Reducers are initialized with Mappers at the job
> initialization, but the reduce method is called in reduce phase when all the
> maps had been finished. So in large jobs where Reducer loads data (>100 MB
> for business logic) in-memory on initialization, the performance can be
> increased by lazily initializing Reducers i.e. loading data in reduce method
> controlled by an initialize flag variable which assures that it is loaded
> only once. By lazily initializing Reducers which require memory (for
> business logic) on initialization, number of maps can be increased.”
>
>
>
> But I did not find any other resource which talks about Reducer Lazy
> initialization.
>
> Does anyone have experience on this ?
>
> If yes, how and where can I set this parameter to get it working.
>
>
>
> Thanks for the support.
>
>
> Regards
> Syed Wasti
>
>
>
+
Arun C Murthy 2010-07-21, 22:22