Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - How to handle schema dependencies


Copy link to this message
-
RE: How to handle schema dependencies
Steven Willis 2012-12-11, 21:42
Thanks Dave,

I was thinking about doing something like that (adding the schemas as resources in the jars). It just seems like a lot of work for something that could be automatic. It would be nice if during schema parsing we could specify a classpath to be used for dynamic lookup of external schemas. That way Schema.parse could look up already schemas that have already been compiled like this:

package com.compete.util;

import org.apache.avro.Schema;
import org.apache.avro.generic.GenericContainer;

public class Schemas {
    public static void main(String args[]) {
        String name = "Date";
        String space = "com.compete.avro";

        try {
            // If we were inside Schema.parse we would probably do:
            // Class.forName(new Name(name, space).toString())

            Class<?> cls = Class.forName(space+"."+name);
            GenericContainer record = (GenericContainer)cls.newInstance();
            Schema schema = record.getSchema();

            // if we were inside Schema.parse(JsonNode schema, Names names) we
            // could now just call: names.put(new Name(name, space), schema);

            System.out.println(schema.toString());
        } catch(ClassNotFoundException e) {
            System.err.println(e);
            System.exit(1);
        } catch(InstantiationException e) {
            System.err.println(e);
            System.exit(1);
        } catch(IllegalAccessException e) {
            System.err.println(e);
            System.exit(1);
        }
    }
}

-Steven Willis

> -----Original Message-----
> From: Dave McAlpin [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, December 11, 2012 3:32 PM
> To: [EMAIL PROTECTED]
> Subject: RE: How to handle schema dependencies
>
> I had the same problem. My solution was to package external schema
> files into a source jar and have Maven download and extract those
> source jars at code generation time. After generation, I delete the
> external schema along with their generated code and depend on an
> external jar file at runtime.
>
> I use IDL instead of Avro schema, so this approach might not work for
> you, but here's what I did.
>
> In the external project (the one I want to import), I changed the
> pom.xml to package schemas into a source jar. Note that the fragment
> below assumes that the Avro schema is stored in src/main/schema, but
> there's nothing special about that location. Also note that I exclude
> generated Java files from the source jar.
>
>    <build>
>       <resources>
>          <resource>
>             <directory>${project.basedir}/src/main/schema</directory>
>          </resource>
>       </resources>
>       <plugins>
>   .
>          .
>          .
>          <plugin>
>             <groupId>org.apache.maven.plugins</groupId>
>             <artifactId>maven-source-plugin</artifactId>
>             <version>2.2.1</version>
>             <executions>
>                <execution>
>                   <id>attach-avdl</id>
>                   <phase>verify</phase>
>                   <goals>
>                      <goal>jar-no-fork</goal>
>                   </goals>
>                   <configuration>
>                      <includePom>true</includePom>
>                      <excludes>
>                         <exclude>**/*.java</exclude>
>                      </excludes>
>                      <includes>
>                         <include>*.avdl</include>
>                      </includes>
>                   </configuration>
>                </execution>
>             </executions>
>          </plugin>
>       </plugins>
>    </build>
>
>
> In the project that uses the external schemas, I changed the pom.xml to
> pull in those schemas as external dependencies and delete them after
> code generation is complete. I also delete the generated java files
> that result from those external schema because I want to use the
> generated class files from an external jar rather than the locally
> generated versions.
>
> ***PLEASE NOTE*** that the code below deletes files as part of clean