I'm trying to write a Reducer which will eliminate duplicates from the list of values before writing them out. I have the following code for my Reducer:

/*****************/
public class ClickStreamIndexerReducer extends Reducer<Text, Text, Text, Text> {
    @Override
    public void reduce(Text dirName, Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context) throws IOException, InterruptedException {
        Text value = new Text();
        Text lastValue = new Text();
        Iterator<Text> valuesIterator = values.iterator();

        while(valuesIterator.hasNext()) {
            value = valuesIterator.next();
            while(value.equals(lastValue)){
                context.write(key, value);
                lastValue = value;
            }
        }
    }
}
/*****************/

Right before the first time "value = valuesIterator.next()" is called, both value and lastValue are empty as expected. Then value is set to the first value and lastValue is still empty. After I write out value I set lastValue to value. The first time through the outer while loop everything goes as expected. However the next time through, when "value = valuesIterator.next()" is called, both value and lastValue are set to the exact same object. Every time through the loop after that, when value is set, lastValue gets set to the same thing.

I seem to have made a few typos in the code. This is how it should read:

/*****************/
public class MyReducer extends Reducer<Text, Text, Text, Text> {
    @Override
    public void reduce(Text key, Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context) throws IOException, InterruptedException {
        Text value = new Text();
        Text lastValue = new Text();
        Iterator<Text> valuesIterator = values.iterator();

        while(valuesIterator.hasNext()) {
            value = valuesIterator.next();
            if(!value.equals(lastValue)){
                context.write(key, value);
                lastValue = value;
            }
        }
    }
}
/*****************/

My description of the problem stands.

-Steven Willis

Writable's are reused.  You need to deep copy the "value" into "lastValue".

On Wed, Mar 14, 2012 at 6:49 PM, Steven Willis <[EMAIL PROTECTED]> wrote:
> I seem to have made a few typos in the code. This is how it should read:
>
> /*****************/
> public class MyReducer extends Reducer<Text, Text, Text, Text> {
>    @Override
>    public void reduce(Text key, Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context) throws IOException, InterruptedException {
>        Text value = new Text();
>        Text lastValue = new Text();
>        Iterator<Text> valuesIterator = values.iterator();
>
>        while(valuesIterator.hasNext()) {
>            value = valuesIterator.next();
>            if(!value.equals(lastValue)){
>                context.write(key, value);
>                lastValue = value;
>            }
>        }
>    }
> }
> /*****************/
>
> My description of the problem stands.
>
> -Steven Willis

--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

What Brock said.

Also read Eric Sammers' response not too long ago on
http://search-hadoop.com/m/Ok1JT1YABHb1 for some more info.

On Thu, Mar 15, 2012 at 6:00 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Writable's are reused.  You need to deep copy the "value" into "lastValue".
>
> On Wed, Mar 14, 2012 at 6:49 PM, Steven Willis <[EMAIL PROTECTED]> wrote:
>> I seem to have made a few typos in the code. This is how it should read:
>>
>> /*****************/
>> public class MyReducer extends Reducer<Text, Text, Text, Text> {
>>    @Override
>>    public void reduce(Text key, Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context) throws IOException, InterruptedException {
>>        Text value = new Text();
>>        Text lastValue = new Text();
>>        Iterator<Text> valuesIterator = values.iterator();
>>
>>        while(valuesIterator.hasNext()) {
>>            value = valuesIterator.next();
>>            if(!value.equals(lastValue)){
>>                context.write(key, value);
>>                lastValue = value;
>>            }
>>        }
>>    }
>> }
>> /*****************/
>>
>> My description of the problem stands.
>>
>> -Steven Willis
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

--
Harsh J

Ah, thank you. Eric's response really cleared that up. So it's the iterator that is re-using the same Text object over and over.

About making the deep copy, Eric writes "If you really need to build a list like this, you'd have to resort to doing a deep copy, but you're better off avoid it if you can as it will drastically impact performance". Would it be more efficient to simply copy the byte array returned from getBytes() to do the comparison?
 
-Steven Willis

-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: Friday, March 16, 2012 3:32 AM
To: [EMAIL PROTECTED]
Subject: Re: Text Iterable modifying other variable

What Brock said.

Also read Eric Sammers' response not too long ago on
http://search-hadoop.com/m/Ok1JT1YABHb1 for some more info.

On Thu, Mar 15, 2012 at 6:00 AM, Brock Noland <[EMAIL PROTECTED]> wrote:
> Writable's are reused.  You need to deep copy the "value" into "lastValue".
>
> On Wed, Mar 14, 2012 at 6:49 PM, Steven Willis <[EMAIL PROTECTED]> wrote:
>> I seem to have made a few typos in the code. This is how it should read:
>>
>> /*****************/
>> public class MyReducer extends Reducer<Text, Text, Text, Text> {
>>    @Override
>>    public void reduce(Text key, Iterable<Text> values, Reducer<Text, Text, Text, Text>.Context context) throws IOException, InterruptedException {
>>        Text value = new Text();
>>        Text lastValue = new Text();
>>        Iterator<Text> valuesIterator = values.iterator();
>>
>>        while(valuesIterator.hasNext()) {
>>            value = valuesIterator.next();
>>            if(!value.equals(lastValue)){
>>                context.write(key, value);
>>                lastValue = value;
>>            }
>>        }
>>    }
>> }
>> /*****************/
>>
>> My description of the problem stands.
>>
>> -Steven Willis
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

--
Harsh J