Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Reading multiple AVSC's with SpecificCompiler


Copy link to this message
-
Reading multiple AVSC's with SpecificCompiler
Lewis John Mcgibbney 2012-12-01, 21:35
Hi,

Over in Apache Goraland, I'm working on imroving the GoraCompiler [0]
(similar in scope to Avro's SpecificCompiler v1.3.3).
Basically, I'm attempting to use the compiler to with multiple avsc's
via an Ant task, however I get the following when attempting to feed
the compiler with multiple schemas.
I've therefore modified the GoraCompiler to compile multiple schemas as follows

  /** Generates Java classes for a schema.
   * @param src the source Avro schema file
   * @param dest the directory to place the generated file in
   */
  public static void compileSchema(File src, File dest) throws IOException {
    compileSchema(new File[] {src}, dest);
  }

  /** Generates Java classes for a number of schema files.
   * @param srcFiles the source Avro schema files
   * @param dest the directory to place the generated files in
   */
  public static void compileSchema(File[] srcFiles, File dest) throws
IOException {
    for(File src : srcFiles) {
      log.info("Compiling schema file: " + src + " to: " + dest );
      GoraCompiler compiler = new GoraCompiler(dest);
      compiler.enqueue(Schema.parse(src));          // enqueue types
      compiler.compile();                           // generate
classes for types
    }
  }
However this gives me the following when I package the source, put it
on my classpath then invoke it through the ant task.

generate-gora-src:
     [java] 12/12/01 21:09:33 INFO compiler.GoraCompiler: Compiling
schema file: /home/lewismc/ASF/2.x/src/gora/host.avsc:/home/lewismc/ASF/2.x/src/gora/webpage.avsc
to: ./src/java
     [java] Exception in thread "main" java.io.FileNotFoundException:
/home/lewismc/ASF/2.x/src/gora/host.avsc:/home/lewismc/ASF/2.x/src/gora/webpage.avsc
(No such file or directory)
     [java]     at java.io.FileInputStream.open(Native Method)
     [java]     at java.io.FileInputStream.<init>(FileInputStream.java:120)
     [java]     at
org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:325)
     [java]     at org.apache.avro.Schema.parse(Schema.java:794)
     [java]     at
org.apache.gora.compiler.GoraCompiler.compileSchema(GoraCompiler.java:89)
     [java]     at
org.apache.gora.compiler.GoraCompiler.compileSchema(GoraCompiler.java:78)
     [java]     at
org.apache.gora.compiler.GoraCompiler.main(GoraCompiler.java:499)
     [java] Java Result: 1

It seems that the ':' file separator does not agree with the code. I
was therefore wondering how Avro's more recent (1.7.x)
SpecificCompiler reads in similar files, and how it deals with
multiple schemas.

Thanks very much in advance for any help.

Lewis

[0] http://svn.apache.org/repos/asf/gora/trunk/gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java

--
Lewis