Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - different output schema when using org.apache.avro.mapred.AvroMultipleOutputs?


Copy link to this message
-
Re: different output schema when using org.apache.avro.mapred.AvroMultipleOutputs?
Pierre Mariani 2013-03-04, 22:08
It looks like what I am looking for, but after applying the patch to
avro mapred 1.7.4 I get the following exception:

java.lang.NullPointerException
         at java.io.StringReader.<init>(StringReader.java:33)
         at org.apache.avro.Schema$Parser.parse(Schema.java:917)
         at org.apache.avro.Schema.parse(Schema.java:966)
         at
org.apache.avro.mapred.AvroMultipleOutputs$InternalFileOutputFormat.getRecordWriter(AvroMultipleOutputs.java:611)
         at
org.apache.avro.mapred.AvroMultipleOutputs.getRecordWriter(AvroMultipleOutputs.java:411)
         at
org.apache.avro.mapred.AvroMultipleOutputs.getCollector(AvroMultipleOutputs.java:570)
         at
org.apache.avro.mapred.AvroMultipleOutputs.getCollector(AvroMultipleOutputs.java:506)
         at
com.inome.diffhadoop.DiffSets$DiffReducer.reduce(DiffSets.java:93)
         at
com.inome.diffhadoop.DiffSets$DiffReducer.reduce(DiffSets.java:40)
         at
org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:60)
         at
org.apache.avro.mapred.HadoopReducerBase.reduce(HadoopReducerBase.java:30)
         at
org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
         at org.apache.hadoop.mapred.Child.main(Child.java:170)
On 03/01/2013 11:07 AM, Doug Cutting wrote:
> Is this the issue addressed by AVRO-1266?
>
> https://issues.apache.org/jira/browse/AVRO-1266
>
> Doug
>
> On Thu, Feb 28, 2013 at 3:11 PM, Pierre Mariani
> <[EMAIL PROTECTED]> wrote:
>> Beside fixing an issue with baseOutputPath,
>> https://issues.apache.org/jira/browse/AVRO-1215 allows to specify different
>> schema for different outputs when using AvroMultipleOutputs.
>>
>> However, this is for the new Hadoop API (org.apache.avro.mapreduce).
>>
>> Is it possible that the same bug exists in the package for the old Hadoop
>> API (org.apache.avro.mapred) ?
>>
>> Thank you
>> Pierre