Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Parsing a Pair's value - inherited namespace?


Copy link to this message
-
Parsing a Pair's value - inherited namespace?
nir_zamir 2013-04-09, 16:02
Hi,

I noticed that after calling:

/AvroJob.setMapOutputSchema(conf,
Pair.getPairSchema(Schema.create(Type.INT), schema));/
(schema is parsed from an avro file, and has no namespace)

When the M/R job is run, there's a call to /AvroJob.getJobOutputSchema
/which calls /Schema.parse/ - which parses the schema I set in
setMapOutputSchema.

The problem: the schema (the value in the Pair returned by
getJobOutputSchema) has no namespace, so it gets it from the Pair (which is
"org.apache.avro.mapred"), so my schema's full name is
"org.apache.avro.mapred.MySchema".

Why do I care? Well, when the schema is a union, the avro items passed to
the AvroMapper.map have no namespace (so their full name is "MySchema", and
are not matching the mapper's output schema (I get a "not in union"
exception). Note that this happens although I pass a Pair to collect(), but
in this case, the Pair's namespace is ignored.

I hope it's clear - will be happy to provide any more details.

Your help is much appreciated!!

Thanks,
Nir
--
View this message in context: http://apache-avro.679487.n3.nabble.com/Parsing-a-Pair-s-value-inherited-namespace-tp4026810.html
Sent from the Avro - Users mailing list archive at Nabble.com.
+
Doug Cutting 2013-04-10, 18:39
+
nir_zamir 2013-04-14, 08:35
+
Doug Cutting 2013-04-16, 23:27
+
nir_zamir 2013-04-18, 07:46
+
nir_zamir 2013-04-25, 12:22