Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> WritableComparable


Copy link to this message
-
Re: WritableComparable
On Sun, Aug 14, 2011 at 9:33 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:

> Does your compareTo() method test object pointer equality? If so, you could
> be getting burned by Hadoop reusing Writable objects.
Yes, but only the equality between enum values.  Interestingly, when
'reduce' is called there are three instances of the "same" key.
Two instances are correctly merged and they both come from the same mapper.
 The other instance comes from a different mapper, and for
some reason does not get merged.  I see the key and the values
(corresponding to the two merged instances) passed as arguments
to 'reduce'; then in subsequent 'reduce' call I see the key and the value
corresponding to the third instance.

For completeness, here is my 'Key.compareTo':

public int compareTo(Key o) {
   if (this.type != o.type) {
       // Type.X < Type.Y
       return (this.type == Type.X ? -1 : 1);
   }
   // otherwise, delegate
   if (this.type == Type.X) {
      return this.key1.compareTo(o.key1);
   } else {
      return this.key2.compareTo(o.key2);
   }
}

The 'type' field is an enum with two possible values, say X and Y.  Key is
essentially a union type; i.e., at any given time
it's the values in key1 or key2 that are being compared (depending on the
'type' value).
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB