-WritableComparable and the case of duplicate keys in the reducer
Stan Rosenberg 2011-08-13, 15:14
Here is what's happening. I have implemented my own WritableComparable keys
Inside a reducer I am seeing 'reduce' being invoked with the "same" key
I have checked that context.getKeyComparator() and
context.getSortComparator() are both WritableComparator which
indicates that 'compareTo' method of my key should be called when doing
Indeed, inside the 'reduce' method I captured both key instances and did the
In both calls, the result is '0', confirming that key1 and key2 are
So, what is going on?
Note that key1 and key2 come from different mappers but they should have
been collapsed in the reducer since
they are both equal according to WritableComparator. Also note that key1
and key2 are not bitwise equivalent, but
that shouldn't matter, or should it?
Many thanks in advance!