Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> WritableComparable


Copy link to this message
-
Re: WritableComparable
What are the types of key1 and key2? What does the readFields() method
look like?

-Joey

On Sun, Aug 14, 2011 at 10:07 PM, Stan Rosenberg
<[EMAIL PROTECTED]> wrote:
> On Sun, Aug 14, 2011 at 9:33 PM, Joey Echeverria <[EMAIL PROTECTED]> wrote:
>
>> Does your compareTo() method test object pointer equality? If so, you could
>> be getting burned by Hadoop reusing Writable objects.
>
>
> Yes, but only the equality between enum values.  Interestingly, when
> 'reduce' is called there are three instances of the "same" key.
> Two instances are correctly merged and they both come from the same mapper.
>  The other instance comes from a different mapper, and for
> some reason does not get merged.  I see the key and the values
> (corresponding to the two merged instances) passed as arguments
> to 'reduce'; then in subsequent 'reduce' call I see the key and the value
> corresponding to the third instance.
>
> For completeness, here is my 'Key.compareTo':
>
> public int compareTo(Key o) {
>   if (this.type != o.type) {
>       // Type.X < Type.Y
>       return (this.type == Type.X ? -1 : 1);
>   }
>   // otherwise, delegate
>   if (this.type == Type.X) {
>      return this.key1.compareTo(o.key1);
>   } else {
>      return this.key2.compareTo(o.key2);
>   }
> }
>
> The 'type' field is an enum with two possible values, say X and Y.  Key is
> essentially a union type; i.e., at any given time
> it's the values in key1 or key2 that are being compared (depending on the
> 'type' value).
>

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB