Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> How to handle schema dependencies


+
Steven Willis 2012-12-11, 19:28
Copy link to this message
-
RE: How to handle schema dependencies
I had the same problem. My solution was to package external schema files into a source jar and have Maven download and extract those source jars at code generation time. After generation, I delete the external schema along with their generated code and depend on an external jar file at runtime.

I use IDL instead of Avro schema, so this approach might not work for you, but here's what I did.

In the external project (the one I want to import), I changed the pom.xml to package schemas into a source jar. Note that the fragment below assumes that the Avro schema is stored in src/main/schema, but there's nothing special about that location. Also note that I exclude generated Java files from the source jar.

   <build>
      <resources>
         <resource>
            <directory>${project.basedir}/src/main/schema</directory>
         </resource>
      </resources>
      <plugins>
  .
         .
         .
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-source-plugin</artifactId>
            <version>2.2.1</version>
            <executions>
               <execution>
                  <id>attach-avdl</id>
                  <phase>verify</phase>
                  <goals>
                     <goal>jar-no-fork</goal>
                  </goals>
                  <configuration>
                     <includePom>true</includePom>
                     <excludes>
                        <exclude>**/*.java</exclude>
                     </excludes>
                     <includes>
                        <include>*.avdl</include>
                     </includes>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>
In the project that uses the external schemas, I changed the pom.xml to pull in those schemas as external dependencies and delete them after code generation is complete. I also delete the generated java files that result from those external schema because I want to use the generated class files from an external jar rather than the locally generated versions.

***PLEASE NOTE*** that the code below deletes files as part of clean up. To use this, you'll need to update the PATH/TO placeholders. If you try this on a real project, MAKE SURE it's backed up before you start testing this.

   <build>
      <plugins>
         .
         .
         .
         <plugin>
            <artifactId>maven-clean-plugin</artifactId>
            <version>2.5</version>
            <executions>
               <execution>
                  <id>clean-generated-java</id>
                  <phase>clean</phase>
                  <goals>
                     <goal>clean</goal>
                  </goals>
                  <configuration>
                     <filesets>
                        <fileset>
                           <directory>src/main/java/PATH/TO/YOUR/GENERATED/DIRECTORY</directory>
                           <includes>
                              <include>*.java</include>
                           </includes>
                        </fileset>
                     </filesets>
                  </configuration>
               </execution>
               <execution>
                  <id>postgen-clean</id>
                  <phase>compile</phase>
                  <goals>
                     <goal>clean</goal>
                  </goals>
                  <configuration>
                     <excludeDefaultDirectories>true</excludeDefaultDirectories>
                     <filesets>
                        <fileset>
                           <directory>src/main/schema</directory>
                           <includes>
                              <include>external/*</include>
                              <include>external</include>
                           </includes>
                        </fileset>
                        <fileset>
                           <directory>src/main/java</directory>
                           <includes>
                              <include>**/*</include>
                           </includes>
                           <excludes>
                              <exclude>PATH/TO/YOUR/GENERATED/SOURCE/DIRECTORY/RELATIVE/TO/SRC/MAIN/JAVA/*</exclude>
                           </excludes>
                        </fileset>
                     </filesets>
                  </configuration>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.5.1</version>
            <executions>
               <execution>
                  <id>import-avdl</id>
                  <phase>initialize</phase>
                  <goals>
                     <goal>unpack-dependencies</goal>
                  </goals>
                  <configuration>
                     <includes>*.avdl</includes>
                     <!-- Limit group ids like this to avoid pulling down source for third party projects
                     <includeGroupIds>com.example,net.example</includeGroupIds>
                     -->
                     <classifier>sources</classifier>
                     <failOnMissingClassifierArtifact>true</failOnMissingClassifierArtifact>
                     <outputDirectory>src/main/schema/external</outputDirectory>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>

In the IDL file that uses these external schemas, I make my imports point to an "external" sub-directory, like this:

   import idl "external/Profile.avdl";

Because I'm pulling all dependent schema files into a single common directory named "external", schema file names need to be unique across all projects. That's not a problem for me, but if it is for you, you could come up with a more sophisticated way to unpack and re
+
Steven Willis 2012-12-11, 21:42
+
Doug Cutting 2012-12-11, 22:54
+
Steven Willis 2012-12-11, 23:47
+
Doug Cutting 2012-12-12, 00:25
+
Dave McAlpin 2012-12-11, 23:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB