Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Combiner function


Copy link to this message
-
Re: Combiner function
On Mon, Aug 2, 2010 at 4:28 PM, Jackob Carlsson
<[EMAIL PROTECTED]> wrote:
> Thanks Nick, but "in-memory" means a combiner can only be used over a single
> mapper?right?! Is there a way we use it for several mappers as well? Also
> what do you mean by "it may or may not run on a particular map attempt"?
>
> Br,
> Jackob
>
> On Mon, Aug 2, 2010 at 5:43 PM, Nick Jones <[EMAIL PROTECTED]> wrote:
>
>> Hi Jackob,
>> A combiner acts a lot like a reduce step but it's executed on the mapper
>> with in-memory data.  I've seen a reduction in job execution time by adding
>> one.  The one caveat to keep in mind is that it may or may not run on a
>> particular map attempt.
>>
>> Nick
>>
>>
>>
>> On 8/2/2010 10:39 AM, Jackob Carlsson wrote:
>>
>>> Hi everyone,
>>> Could anyone please help me to understand the function of combiner?
>>>
>>> Thanks in advance
>>> Jackob
>>>
>>>
>>
>>
>

> Is there a way we use it for several mappers as well?
No. That is the exact opposite goal of the combiner. It runs locally.
>it may or may not run on a particular map attempt
It only runs when certain thresholds in the framework are reached.

http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/