Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> race condition in hadoop 0.20.2 (cdh3u1)


+
Stan Rosenberg 2012-01-17, 21:21
Copy link to this message
-
Re: race condition in hadoop 0.20.2 (cdh3u1)
Hi,

tl;dr DUMMY should not be static.

On Tue, Jan 17, 2012 at 3:21 PM, Stan Rosenberg
<[EMAIL PROTECTED]> wrote:
>
>
> class MyKey<T> implements WritableComparable<T> {
>  private String ip; // first part of the key
>  private final static Text DUMMY = new Text();
>  ...
>
>  public void write(DataOutput out) throws IOException {
>     // serialize the first part of the key
>     DUMMY.set(ip);
>     DUMMY.write(out);
>     ...
>  }
>
>  public void readFields(DataInput in) throws IOException {
>    // de-serialize the first part of the key
>    DUMMY.readFields(in); ip = DUMMY.toString();
>    ....
>  }
> }

This class is invalid. A single thread will be executing your mapper
or reducer but there will be multiple threads (background threads such
as the SpillThread) creating MyKey instances which is exactly what you
are seeing. This is by design.

Brock
+
Stan Rosenberg 2012-01-17, 23:51