Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to handle schema dependencies


Copy link to this message
-
RE: How to handle schema dependencies
That enhancement allows my approach to be used with avsc files as well as IDL files.

Here's our use case.

Project A defines the IDL for a record of type A. Something like this:

**File a.avdl
@namespace("com.example.a")
protocol AService {
   record A {
     union {null, string} aData = null;
}

Project A provides a jar file with the generated Java classes for A.

Project B creates an IDL that uses type A. Something like this:

**File b.avdl
@namespace("com.example.b")
protocol BService {
   import idl "a.avdl"
   record B {
     union {null, com.example.a.A} a = null;
}

Ideally, Project B doesn't need access to Project A's source code; it will depend only on the jar file from Project A.

There are two problems. First, the Avro code generator in Project B needs access to a.avdl in order to satisfy the import in b.avdl, but a.avdl is only available in project A. Second, the Avro code generator will generate Java classes for A in Project B, which duplicate the Java files already available in the jar file in Project A.

My solution is to have Maven package up a.avdl in Project A's source jar. I then add some Maven-ness to project B to do the following at code generation time.

1) pull down the sources jar from Project A
2) unpack any avdl files from Project A's sources jar into a "external" directory under Project B's schema directory
3) run the Avro generator on Project B's schema
4) delete the "external" directory and its contents
5) delete any Java files generated as a result of the avdl files in Project B's "external" directory

This allows Project B to do something like 'import idl "external/a.avdl"' and have that external dependency satisfied by Maven at code generation time without having explicit access to Project A's source.

Dave

-----Original Message-----
From: Doug Cutting [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, December 11, 2012 2:55 PM
To: [EMAIL PROTECTED]
Subject: Re: How to handle schema dependencies

On Tue, Dec 11, 2012 at 1:42 PM, Steven Willis <[EMAIL PROTECTED]> wrote:
> It would be nice if during schema parsing we could specify a classpath
> to be used for dynamic lookup of external schemas

Does AVRO-1188 (included in Avro 1.7.3) help here?

https://issues.apache.org/jira/browse/AVRO-1188

This permits one to specify directories of schemas to import to Avro Maven executions.

Doug
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB