Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Problem with Reducer emitting a different Key than Mapper


Copy link to this message
-
Problem with Reducer emitting a different Key than Mapper
Problem with Reducer emitting a different Key than Mapper
IO have the FOllowing code where the Mapper emits a custom Key and the
reducer isa expected to emit text

Using Hadoop 0.2 on a local instance  I asj the reducer to write Text,Text -
thiew is even what the IDE says I should do
and what I get is the exception below - any bright ideas???

public class PartitionReducer extends Reducer<GenonePartitionKey, Text,
Text, Text>
{

  /**
     * This method is called once for each key. Most applications will
define
     * their reduce class by overriding this method. The default
implementation
     * is an identity function.
     */
    @Override
    protected void reduce(GenonePartitionKey key, Iterable<Text> values,
Context context)
            throws IOException, InterruptedException
    {
          context.write(new Text("Foo"),new Text("Bar"));

     ....
 }

 }
10/06/16 09:08:45 INFO mapred.MapTask: Starting flush of map output
10/06/16 09:08:45 WARN mapred.LocalJobRunner: job_local_0001
java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1123)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177)
Caused by: java.io.IOException: wrong key class: class
org.apache.hadoop.io.Text is not class
org.systemsbiology.hadoop.GenonePartitionKey
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:164)
at
org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:880)
at
org.apache.hadoop.mapred.Task$NewCombinerRunner$OutputConverter.write(Task.java:1201)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at
org.systemsbiology.hadoop.PartitionReducer.reduce(PartitionReducer.java:165)
at
org.systemsbiology.hadoop.PartitionReducer.reduce(PartitionReducer.java:23)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
at org.systemsbiology.hadoop.PartitionReducer.run(PartitionReducer.java:259)
at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1222)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1265)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:686)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1173)
--
Steven M. Lewis PhD
Institute for Systems Biology
Seattle WA
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB