i am currently testing my map reduce job on Windows + Cygwin + Hadoop
v0.20.205. for some strange reason, the list of values (i.e.
Iterable<T> values) going into the reducer looks all wrong. i have
tracked the map reduce process with logging statements (i.e. logged
the input to the map, logged the output from the map, logged the
partitioner, logged the input to the reducer). at all stages,
everything looks correct except at the reducer.

is there anyway (using Windows  + Cygwin) to view the local map
outputs before they are shuffled/sorted to the reducer? i need to know
why the values are incorrect.

i found out what my problem was. apparently, when you iterate over
Iterable<Type> values, that instance of Type is being used over and
over. for example, in my reducer,

public void reduce(Key key, Iterator<Value> values, Context context)
throws IOException, InterruptedException {
 Iterator<Value> it = values.iterator();
 Value a = it.next();
 Value b = it.next();
}

the variables, a and b of type Value, will be the same object
instance! i suppose this behavior of the iterator is to optimize
iterating so as to avoid the new operator.

On Thu, Apr 5, 2012 at 4:55 PM, Jane Wayne <[EMAIL PROTECTED]> wrote:
> i am currently testing my map reduce job on Windows + Cygwin + Hadoop
> v0.20.205. for some strange reason, the list of values (i.e.
> Iterable<T> values) going into the reducer looks all wrong. i have
> tracked the map reduce process with logging statements (i.e. logged
> the input to the map, logged the output from the map, logged the
> partitioner, logged the input to the reducer). at all stages,
> everything looks correct except at the reducer.
>
> is there anyway (using Windows  + Cygwin) to view the local map
> outputs before they are shuffled/sorted to the reducer? i need to know
> why the values are incorrect.

Jane,

Yes and thats documented:
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Reducer.html#reduce(K2,%20java.util.Iterator,%20org.apache.hadoop.mapred.OutputCollector,%20org.apache.hadoop.mapred.Reporter)

"The framework will reuse the key and value objects that are passed
into the reduce, therefore the application should clone the objects
they want to keep a copy of."

On Fri, Apr 6, 2012 at 6:26 AM, Jane Wayne <[EMAIL PROTECTED]> wrote:
> i found out what my problem was. apparently, when you iterate over
> Iterable<Type> values, that instance of Type is being used over and
> over. for example, in my reducer,
>
> public void reduce(Key key, Iterator<Value> values, Context context)
> throws IOException, InterruptedException {
>  Iterator<Value> it = values.iterator();
>  Value a = it.next();
>  Value b = it.next();
> }
>
> the variables, a and b of type Value, will be the same object
> instance! i suppose this behavior of the iterator is to optimize
> iterating so as to avoid the new operator.
>
>
>
> On Thu, Apr 5, 2012 at 4:55 PM, Jane Wayne <[EMAIL PROTECTED]> wrote:
>> i am currently testing my map reduce job on Windows + Cygwin + Hadoop
>> v0.20.205. for some strange reason, the list of values (i.e.
>> Iterable<T> values) going into the reducer looks all wrong. i have
>> tracked the map reduce process with logging statements (i.e. logged
>> the input to the map, logged the output from the map, logged the
>> partitioner, logged the input to the reducer). at all stages,
>> everything looks correct except at the reducer.
>>
>> is there anyway (using Windows  + Cygwin) to view the local map
>> outputs before they are shuffled/sorted to the reducer? i need to know
>> why the values are incorrect.

--
Harsh J