Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> How to handle schema dependencies


+
Steven Willis 2012-12-11, 19:28
Copy link to this message
-
RE: How to handle schema dependencies
I had the same problem. My solution was to package external schema files into a source jar and have Maven download and extract those source jars at code generation time. After generation, I delete the external schema along with their generated code and depend on an external jar file at runtime.

I use IDL instead of Avro schema, so this approach might not work for you, but here's what I did.

In the external project (the one I want to import), I changed the pom.xml to package schemas into a source jar. Note that the fragment below assumes that the Avro schema is stored in src/main/schema, but there's nothing special about that location. Also note that I exclude generated Java files from the source jar.

   <build>
      <resources>
         <resource>
            <directory>${project.basedir}/src/main/schema</directory>
         </resource>
      </resources>
      <plugins>
  .
         .
         .
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-source-plugin</artifactId>
            <version>2.2.1</version>
            <executions>
               <execution>
                  <id>attach-avdl</id>
                  <phase>verify</phase>
                  <goals>
                     <goal>jar-no-fork</goal>
                  </goals>
                  <configuration>
                     <includePom>true</includePom>
                     <excludes>
                        <exclude>**/*.java</exclude>
                     </excludes>
                     <includes>
                        <include>*.avdl</include>
                     </includes>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
In the project that uses the external schemas, I changed the pom.xml to pull in those schemas as external dependencies and delete them after code generation is complete. I also delete the generated java files that result from those external schema because I want to use the generated class files from an external jar rather than the locally generated versions.

***PLEASE NOTE*** that the code below deletes files as part of clean up. To use this, you'll need to update the PATH/TO placeholders. If you try this on a real project, MAKE SURE it's backed up before you start testing this.

   <build>
      <plugins>
         .
         .
         .
         <plugin>
            <artifactId>maven-clean-plugin</artifactId>
            <version>2.5</version>
            <executions>
               <execution>
                  <id>clean-generated-java</id>
                  <phase>clean</phase>
                  <goals>
                     <goal>clean</goal>
                  </goals>
                  <configuration>
                     <filesets>
                        <fileset>
                           <directory>src/main/java/PATH/TO/YOUR/GENERATED/DIRECTORY</directory>
                           <includes>
                              <include>*.java</include>
                           </includes>
                        </fileset>
                     </filesets>
                  </configuration>
               </execution>
               <execution>
                  <id>postgen-clean</id>
                  <phase>compile</phase>
                  <goals>
                     <goal>clean</goal>
                  </goals>
                  <configuration>
                     <excludeDefaultDirectories>true</excludeDefaultDirectories>
                     <filesets>
                        <fileset>
                           <directory>src/main/schema</directory>
                           <includes>
                              <include>external/*</include>
                              <include>external</include>
                           </includes>
                        </fileset>
                        <fileset>
                           <directory>src/main/java</directory>
                           <includes>
                              <include>**/*</include>
                           </includes>
                           <excludes>
                              <exclude>PATH/TO/YOUR/GENERATED/SOURCE/DIRECTORY/RELATIVE/TO/SRC/MAIN/JAVA/*</exclude>
                           </excludes>
                        </fileset>
                     </filesets>
                  </configuration>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.5.1</version>
            <executions>
               <execution>
                  <id>import-avdl</id>
                  <phase>initialize</phase>
                  <goals>
                     <goal>unpack-dependencies</goal>
                  </goals>
                  <configuration>
                     <includes>*.avdl</includes>
                     <!-- Limit group ids like this to avoid pulling down source for third party projects
                     <includeGroupIds>com.example,net.example</includeGroupIds>
                     -->
                     <classifier>sources</classifier>
                     <failOnMissingClassifierArtifact>true</failOnMissingClassifierArtifact>
                     <outputDirectory>src/main/schema/external</outputDirectory>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>

In the IDL file that uses these external schemas, I make my imports point to an "external" sub-directory, like this:

   import idl "external/Profile.avdl";

Because I'm pulling all dependent schema files into a single common directory named "external", schema file names need to be unique across all projects. That's not a problem for me, but if it is for you, you could come up with a more sophisticated way to unpack and re
+
Steven Willis 2012-12-11, 21:42
+
Doug Cutting 2012-12-11, 22:54
+
Steven Willis 2012-12-11, 23:47
+
Doug Cutting 2012-12-12, 00:25
+
Dave McAlpin 2012-12-11, 23:45