Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Combiner function


Copy link to this message
-
Re: Combiner function
On Mon, Aug 2, 2010 at 4:28 PM, Jackob Carlsson
<[EMAIL PROTECTED]> wrote:
> Thanks Nick, but "in-memory" means a combiner can only be used over a single
> mapper?right?! Is there a way we use it for several mappers as well? Also
> what do you mean by "it may or may not run on a particular map attempt"?
>
> Br,
> Jackob
>
> On Mon, Aug 2, 2010 at 5:43 PM, Nick Jones <[EMAIL PROTECTED]> wrote:
>
>> Hi Jackob,
>> A combiner acts a lot like a reduce step but it's executed on the mapper
>> with in-memory data.  I've seen a reduction in job execution time by adding
>> one.  The one caveat to keep in mind is that it may or may not run on a
>> particular map attempt.
>>
>> Nick
>>
>>
>>
>> On 8/2/2010 10:39 AM, Jackob Carlsson wrote:
>>
>>> Hi everyone,
>>> Could anyone please help me to understand the function of combiner?
>>>
>>> Thanks in advance
>>> Jackob
>>>
>>>
>>
>>
>

> Is there a way we use it for several mappers as well?
No. That is the exact opposite goal of the combiner. It runs locally.
>it may or may not run on a particular map attempt
It only runs when certain thresholds in the framework are reached.

http://philippeadjiman.com/blog/2010/01/14/hadoop-tutorial-series-issue-4-to-use-or-not-to-use-a-combiner/
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB