Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - WritableComparable


Copy link to this message
-
Re: WritableComparable
Joey Echeverria 2011-08-15, 02:25
What are the types of key1 and key2? What does the readFields() method
look like?

-Joey

On Sun, Aug 14, 2011 at 10:07 PM, Stan Rosenberg
<[EMAIL PROTECTED]> wrote:
> On Sun, Aug 14, 2011 at 9:33 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:
>
>> Does your compareTo() method test object pointer equality? If so, you could
>> be getting burned by Hadoop reusing Writable objects.
>
>
> Yes, but only the equality between enum values.  Interestingly, when
> 'reduce' is called there are three instances of the "same" key.
> Two instances are correctly merged and they both come from the same mapper.
>  The other instance comes from a different mapper, and for
> some reason does not get merged.  I see the key and the values
> (corresponding to the two merged instances) passed as arguments
> to 'reduce'; then in subsequent 'reduce' call I see the key and the value
> corresponding to the third instance.
>
> For completeness, here is my 'Key.compareTo':
>
> public int compareTo(Key o) {
>   if (this.type != o.type) {
>       // Type.X < Type.Y
>       return (this.type == Type.X ? -1 : 1);
>   }
>   // otherwise, delegate
>   if (this.type == Type.X) {
>      return this.key1.compareTo(o.key1);
>   } else {
>      return this.key2.compareTo(o.key2);
>   }
> }
>
> The 'type' field is an enum with two possible values, say X and Y.  Key is
> essentially a union type; i.e., at any given time
> it's the values in key1 or key2 that are being compared (depending on the
> 'type' value).
>

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434