Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> race condition in hadoop 0.20.2 (cdh3u1)


Copy link to this message
-
Re: race condition in hadoop 0.20.2 (cdh3u1)
Hi,

tl;dr DUMMY should not be static.

On Tue, Jan 17, 2012 at 3:21 PM, Stan Rosenberg
<[EMAIL PROTECTED]> wrote:
>
>
> class MyKey<T> implements WritableComparable<T> {
>  private String ip; // first part of the key
>  private final static Text DUMMY = new Text();
>  ...
>
>  public void write(DataOutput out) throws IOException {
>     // serialize the first part of the key
>     DUMMY.set(ip);
>     DUMMY.write(out);
>     ...
>  }
>
>  public void readFields(DataInput in) throws IOException {
>    // de-serialize the first part of the key
>    DUMMY.readFields(in); ip = DUMMY.toString();
>    ....
>  }
> }

This class is invalid. A single thread will be executing your mapper
or reducer but there will be multiple threads (background threads such
as the SpillThread) creating MyKey instances which is exactly what you
are seeing. This is by design.

Brock
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB