Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> AvroMapper map method not getting invoked


Copy link to this message
-
Re: AvroMapper map method not getting invoked
One thing I've run into is not properly overriding the map and reduce
method.

The AvroMapper (
http://svn.apache.org/viewvc/avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroMapper.java?view=markup
)
defines a default implementation for map. So if you haven't properly
overloaded this method then your map function won't get invoked and you'll
wonder why nothing is happening when really the default implementation is
being invoked and it does nothing.

An easy way to check this by adding @Override to your map function. If your
method doesn't match an inherited function this should cause a compiler
warning.

J

On Thu, Mar 29, 2012 at 1:42 PM, Serge Blazhievsky <[EMAIL PROTECTED]>wrote:

> Can you post the code for your main method?
>
> Serge
>
>
> On Thu, Mar 29, 2012 at 12:54 PM, Deepak Nettem <[EMAIL PROTECTED]>wrote:
>
>> HI,
>>
>> I am trying to run a simple Avro MapReduce Mapper only job, that reads
>> from an Avro source. This piece of code simply reads my avro source data
>> and writes it to the outpath in the exact same schema, irrespective of what
>> is given in the map method of the mapper. The map method is not getting
>> invoked at all.
>>
>> Why is this happening?
>>
>> -----------------
>> public class AvroReader {
>>
>>     //public static final Schema schema;
>>     public static final Schema IN_SCHEMA >> ReflectData.get().getSchema(Record.class);
>>
>>   public void AvroReaderRunner(String inputPath, String outputPath)
>> throws Exception {
>>
>>    JobConf conf = new JobConf();
>>
>>    Job job = new Job(conf,"AvroRecord Job..");
>>     job.setJobName("avroreader");
>>     job.setJarByClass(AvroReader.class);
>>
>>     AvroJob.setInputSchema(conf, IN_SCHEMA);
>>
>>     //Schema stringSchema = Schema.create(Schema.Type.STRING);
>>     //Schema pairSchema = Pair.getPairSchema(stringSchema, stringSchema);
>>
>>     AvroJob.setMapOutputSchema(conf, new Pair<Utf8,Utf8>(new Utf8(""),
>> 0L).getSchema() );
>>
>>     //AvroJob.setOutputSchema(conf, IN_SCHEMA.)
>>     AvroJob.setMapperClass(conf, AvroReaderMapper.class);
>>
>>     FileInputFormat.addInputPath(job, new Path(inputPath));
>>
>>     //FileInputFormat.addInputPath(job, new Path(inputPath));
>>     FileOutputFormat.setOutputPath(job, new Path(outputPath));
>>
>>     //FileOutputFormat.setOutputPath(job, new Path(outputPath));
>>
>>     job.setNumReduceTasks(0);                     // map-only
>>
>>     job.waitForCompletion(true);
>>     }
>>
>>   public static class AvroReaderMapper extends AvroMapper<Record,
>> Pair<Utf8, Utf8>>{
>>
>>       @Override
>>     public void map(Record in, AvroCollector<Pair<Utf8,Utf8>> collector,
>>                       Reporter reporter) throws IOException {
>>         collector.collect(new Pair<Utf8,Utf8>('a', 'b'));
>>
>>     }
>>
>>   }
>>
>> }
>>
>>
>> --
>> Warm Regards,
>> Deepak Nettem
>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB