Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to handle schema dependencies


Copy link to this message
-
RE: How to handle schema dependencies
Thanks Dave,

I was thinking about doing something like that (adding the schemas as resources in the jars). It just seems like a lot of work for something that could be automatic. It would be nice if during schema parsing we could specify a classpath to be used for dynamic lookup of external schemas. That way Schema.parse could look up already schemas that have already been compiled like this:

package com.compete.util;

import org.apache.avro.Schema;
import org.apache.avro.generic.GenericContainer;

public class Schemas {
    public static void main(String args[]) {
        String name = "Date";
        String space = "com.compete.avro";

        try {
            // If we were inside Schema.parse we would probably do:
            // Class.forName(new Name(name, space).toString())

            Class<?> cls = Class.forName(space+"."+name);
            GenericContainer record = (GenericContainer)cls.newInstance();
            Schema schema = record.getSchema();

            // if we were inside Schema.parse(JsonNode schema, Names names) we
            // could now just call: names.put(new Name(name, space), schema);

            System.out.println(schema.toString());
        } catch(ClassNotFoundException e) {
            System.err.println(e);
            System.exit(1);
        } catch(InstantiationException e) {
            System.err.println(e);
            System.exit(1);
        } catch(IllegalAccessException e) {
            System.err.println(e);
            System.exit(1);
        }
    }
}

-Steven Willis

> -----Original Message-----
> From: Dave McAlpin [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, December 11, 2012 3:32 PM
> To: [EMAIL PROTECTED]
> Subject: RE: How to handle schema dependencies
>
> I had the same problem. My solution was to package external schema
> files into a source jar and have Maven download and extract those
> source jars at code generation time. After generation, I delete the
> external schema along with their generated code and depend on an
> external jar file at runtime.
>
> I use IDL instead of Avro schema, so this approach might not work for
> you, but here's what I did.
>
> In the external project (the one I want to import), I changed the
> pom.xml to package schemas into a source jar. Note that the fragment
> below assumes that the Avro schema is stored in src/main/schema, but
> there's nothing special about that location. Also note that I exclude
> generated Java files from the source jar.
>
>    <build>
>       <resources>
>          <resource>
>             <directory>${project.basedir}/src/main/schema</directory>
>          </resource>
>       </resources>
>       <plugins>
>   .
>          .
>          .
>          <plugin>
>             <groupId>org.apache.maven.plugins</groupId>
>             <artifactId>maven-source-plugin</artifactId>
>             <version>2.2.1</version>
>             <executions>
>                <execution>
>                   <id>attach-avdl</id>
>                   <phase>verify</phase>
>                   <goals>
>                      <goal>jar-no-fork</goal>
>                   </goals>
>                   <configuration>
>                      <includePom>true</includePom>
>                      <excludes>
>                         <exclude>**/*.java</exclude>
>                      </excludes>
>                      <includes>
>                         <include>*.avdl</include>
>                      </includes>
>                   </configuration>
>                </execution>
>             </executions>
>          </plugin>
>       </plugins>
>    </build>
>
>
> In the project that uses the external schemas, I changed the pom.xml to
> pull in those schemas as external dependencies and delete them after
> code generation is complete. I also delete the generated java files
> that result from those external schema because I want to use the
> generated class files from an external jar rather than the locally
> generated versions.
>
> ***PLEASE NOTE*** that the code below deletes files as part of clean
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB