Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to handle schema dependencies


Copy link to this message
-
How to handle schema dependencies
Hi,

My company currently has one big repo that holds all our java code and avro schemas. I'm currently splitting it up into one common repo and separate repos for each product. This has been easy to do for our java code using maven and dependencies, however I can't find a way to do this with our avro schemas. In the common repo I've got schemas that everything relies on. These common schemas include some very domain specific stuff along with some pretty general use schemas that we've defined like Date:

 {"name": "Date",
  "namespace": "com.compete.avro",
  "type": "record",
  "fields": [ { "name":"year", "type":"int" },
              { "name":"month", "type":"int" },
              { "name":"day", "type":"int" } ] }
I'd like to be able to use 'Date' (defined in the common repo) in schemas inside the product-xyz repo. But when I try this, I get:

[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] "Date" is not a defined name. The type of the "date" field must be a defined name or a {"type": ...} expression.
[INFO] ------------------------------------------------------------------------
[INFO] Trace
org.apache.avro.SchemaParseException: "Date" is not a defined name. The type of the "date" field must be a defined name or a {"type": ...} expression.
at org.apache.avro.Schema.parse(Schema.java:1094)
at org.apache.avro.Schema.parse(Schema.java:1163)
at org.apache.avro.Schema$Parser.parse(Schema.java:931)
at org.apache.avro.Schema$Parser.parse(Schema.java:908)
at org.apache.avro.compiler.specific.SpecificCompiler.compileSchema(SpecificCompiler.java:182)
at org.apache.avro.compiler.specific.SpecificCompiler.compileSchema(SpecificCompiler.java:174)
at org.apache.avro.mojo.SchemaMojo.doCompile(SchemaMojo.java:53)
at org.apache.avro.mojo.AbstractAvroMojo.compileFiles(AbstractAvroMojo.java:129)
at org.apache.avro.mojo.AbstractAvroMojo.execute(AbstractAvroMojo.java:99)
at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLifecycle(DefaultLifecycleExecutor.java:556)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:535)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)
at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
at org.codehaus.classworlds.Launcher.main(Launcher.java:375)

The common jar which contains the compiled Date class is available in our maven repo... is there some way to use that? I'm currently using the avro-maven-plugin to do the java code generation, is there an option to this plugin to specify schemas or jars to include? It seems like the only work around is to put all avro schemas that we might use in any product in the common repo, or duplicate all the common schemas inside each product specific repo.

-Steven Willis
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB