Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Force job to use all reducers evenly?


Copy link to this message
-
Re: Force job to use all reducers evenly?
Keith Wiley 2011-03-25, 14:18
On Mar 25, 2011, at 2:02 AM, Harsh J wrote:

> On Fri, Mar 25, 2011 at 12:48 PM, Keith Wiley <[EMAIL PROTECTED]> wrote:
>> Say my mappers produce at most (or precisely) 4 output keys.  Say I designate the job to have at least (or precisely) 4 reducers.  I have noticed that it is not guaranteed that all four reducers will be used, one per key.  Rather, it is entirely likely that one reducer won't be used at all and another will receive two sets of keys, first receiving all values of one key, then all values of the other key.
>
> This is merely the side effect of using a hash-based partitioner when

Ah, of course.  I'm know how the hash partitioners work, I just didn't realize (ok, remember) that the default partitioner works that way.

________________________________________________________________________________
Keith Wiley     [EMAIL PROTECTED]     keithwiley.com    music.keithwiley.com

"I used to be with it, but then they changed what it was.  Now, what I'm with
isn't it, and what's it seems weird and scary to me."
                                           --  Abe (Grandpa) Simpson
________________________________________________________________________________