Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> How to handle schema dependencies

Steven Willis 2012-12-11, 19:28
Dave McAlpin 2012-12-11, 20:31
Steven Willis 2012-12-11, 21:42
Doug Cutting 2012-12-11, 22:54
Steven Willis 2012-12-11, 23:47
Doug Cutting 2012-12-12, 00:25
Copy link to this message
RE: How to handle schema dependencies
That enhancement allows my approach to be used with avsc files as well as IDL files.

Here's our use case.

Project A defines the IDL for a record of type A. Something like this:

**File a.avdl
protocol AService {
   record A {
     union {null, string} aData = null;

Project A provides a jar file with the generated Java classes for A.

Project B creates an IDL that uses type A. Something like this:

**File b.avdl
protocol BService {
   import idl "a.avdl"
   record B {
     union {null, com.example.a.A} a = null;

Ideally, Project B doesn't need access to Project A's source code; it will depend only on the jar file from Project A.

There are two problems. First, the Avro code generator in Project B needs access to a.avdl in order to satisfy the import in b.avdl, but a.avdl is only available in project A. Second, the Avro code generator will generate Java classes for A in Project B, which duplicate the Java files already available in the jar file in Project A.

My solution is to have Maven package up a.avdl in Project A's source jar. I then add some Maven-ness to project B to do the following at code generation time.

1) pull down the sources jar from Project A
2) unpack any avdl files from Project A's sources jar into a "external" directory under Project B's schema directory
3) run the Avro generator on Project B's schema
4) delete the "external" directory and its contents
5) delete any Java files generated as a result of the avdl files in Project B's "external" directory

This allows Project B to do something like 'import idl "external/a.avdl"' and have that external dependency satisfied by Maven at code generation time without having explicit access to Project A's source.


-----Original Message-----
From: Doug Cutting [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, December 11, 2012 2:55 PM
Subject: Re: How to handle schema dependencies

On Tue, Dec 11, 2012 at 1:42 PM, Steven Willis <[EMAIL PROTECTED]> wrote:
> It would be nice if during schema parsing we could specify a classpath
> to be used for dynamic lookup of external schemas

Does AVRO-1188 (included in Avro 1.7.3) help here?


This permits one to specify directories of schemas to import to Avro Maven executions.