Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - error:Type mismatch in value from map


Copy link to this message
-
Re: error:Type mismatch in value from map
Joey Echeverria 2011-07-29, 14:49
If you want to use a combiner, your map has to output the same types
as your combiner outputs. In your case, modify your map to look like
this:

  public static class TokenizerMapper
       extends Mapper<Text, Text, Text, IntWritable>{
    public void map(Text key, Text value, Context context
                    ) throws IOException, InterruptedException {
        context.write(key, new IntWritable(1));
    }
  }

>  11/07/29 22:22:22 INFO mapred.JobClient: Task Id : attempt_201107292131_0011_m_000000_2, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text
>
> But I already set IntWritable in 2 places,
> 1: Reducer<Text,Text,Text,IntWritable>
> 2:job.setOutputValueClass(IntWritable.class);
>
> So where am I wrong?
>
> public class MyTest {
>
>  public static class TokenizerMapper
>       extends Mapper<Text, Text, Text, Text>{
>    public void map(Text key, Text value, Context context
>                    ) throws IOException, InterruptedException {
>        context.write(key, value);
>    }
>  }
>
>  public static class IntSumReducer
>       extends Reducer<Text,Text,Text,IntWritable> {
>
>    public void reduce(Text key, Iterable<Text> values,
>                       Context context
>                       ) throws IOException, InterruptedException {
>       int count = 0;
>       for (Text iw:values) {
>            count++;
>       }
>      context.write(key, new IntWritable(count));
>     }
>   }
>
>  public static void main(String[] args) throws Exception {
>    Configuration conf = new Configuration();
> // the configure of seprator should be done in conf
>    conf.set("key.value.separator.in.input.line", ",");
>    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
>    if (otherArgs.length != 2) {
>      System.err.println("Usage: wordcount <in> <out>");
>      System.exit(2);
>    }
>    Job job = new Job(conf, "word count");
>    job.setJarByClass(WordCount.class);
>    job.setMapperClass(TokenizerMapper.class);
>    job.setCombinerClass(IntSumReducer.class);
> //    job.setReducerClass(IntSumReducer.class);
>    job.setInputFormatClass(KeyValueTextInputFormat.class);
>    // job.set("key.value.separator.in.input.line", ",");
>    job.setOutputKeyClass(Text.class);
>    job.setOutputValueClass(IntWritable.class);
>    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
>    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
>    System.exit(job.waitForCompletion(true) ? 0 : 1);
>  }
> }
>

--
Joseph Echeverria
Cloudera, Inc.
443.305.9434