Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> How to handle schema dependencies

Copy link to this message
How to handle schema dependencies

My company currently has one big repo that holds all our java code and avro schemas. I'm currently splitting it up into one common repo and separate repos for each product. This has been easy to do for our java code using maven and dependencies, however I can't find a way to do this with our avro schemas. In the common repo I've got schemas that everything relies on. These common schemas include some very domain specific stuff along with some pretty general use schemas that we've defined like Date:

 {"name": "Date",
  "namespace": "com.compete.avro",
  "type": "record",
  "fields": [ { "name":"year", "type":"int" },
              { "name":"month", "type":"int" },
              { "name":"day", "type":"int" } ] }
I'd like to be able to use 'Date' (defined in the common repo) in schemas inside the product-xyz repo. But when I try this, I get:

[INFO] ------------------------------------------------------------------------
[INFO] "Date" is not a defined name. The type of the "date" field must be a defined name or a {"type": ...} expression.
[INFO] ------------------------------------------------------------------------
[INFO] Trace
org.apache.avro.SchemaParseException: "Date" is not a defined name. The type of the "date" field must be a defined name or a {"type": ...} expression.
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema.parse(Schema.java:1163)
at org.apache.avro.Schema$Parser.parse(Schema.java:931)
at org.apache.avro.Schema$Parser.parse(Schema.java:908)
at org.apache.avro.compiler.specific.SpecificCompiler.compileSchema(SpecificCompiler.java:182)
at org.apache.avro.compiler.specific.SpecificCompiler.compileSchema(SpecificCompiler.java:174)
at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:53)
at org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:129)
at org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:99)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLifecycle(DefaultLifecycleExecutor.java:556)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:535)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)

The common jar which contains the compiled Date class is available in our maven repo... is there some way to use that? I'm currently using the avro-maven-plugin to do the java code generation, is there an option to this plugin to specify schemas or jars to include? It seems like the only work around is to put all avro schemas that we might use in any product in the common repo, or duplicate all the common schemas inside each product specific repo.

-Steven Willis
Dave McAlpin 2012-12-11, 20:31
Steven Willis 2012-12-11, 21:42
Doug Cutting 2012-12-11, 22:54
Steven Willis 2012-12-11, 23:47
Doug Cutting 2012-12-12, 00:25
Dave McAlpin 2012-12-11, 23:45