Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Reading multiple AVSC's with SpecificCompiler


Copy link to this message
-
Reading multiple AVSC's with SpecificCompiler
Hi,

Over in Apache Goraland, I'm working on imroving the GoraCompiler [0]
(similar in scope to Avro's SpecificCompiler v1.3.3).
Basically, I'm attempting to use the compiler to with multiple avsc's
via an Ant task, however I get the following when attempting to feed
the compiler with multiple schemas.
I've therefore modified the GoraCompiler to compile multiple schemas as follows

  /** Generates Java classes for a schema.
   * @param src the source Avro schema file
   * @param dest the directory to place the generated file in
   */
  public static void compileSchema(File src, File dest) throws IOException {
    compileSchema(new File[] {src}, dest);
  }

  /** Generates Java classes for a number of schema files.
   * @param srcFiles the source Avro schema files
   * @param dest the directory to place the generated files in
   */
  public static void compileSchema(File[] srcFiles, File dest) throws
IOException {
    for(File src : srcFiles) {
      log.info("Compiling schema file: " + src + " to: " + dest );
      GoraCompiler compiler = new GoraCompiler(dest);
      compiler.enqueue(Schema.parse(src));          // enqueue types
      compiler.compile();                           // generate
classes for types
    }
  }
However this gives me the following when I package the source, put it
on my classpath then invoke it through the ant task.

generate-gora-src:
     [java] 12/12/01 21:09:33 INFO compiler.GoraCompiler: Compiling
schema file: /home/lewismc/ASF/2.x/src/gora/host.avsc:/home/lewismc/ASF/2.x/src/gora/webpage.avsc
to: ./src/java
     [java] Exception in thread "main" java.io.FileNotFoundException:
/home/lewismc/ASF/2.x/src/gora/host.avsc:/home/lewismc/ASF/2.x/src/gora/webpage.avsc
(No such file or directory)
     [java]     at java.io.FileInputStream.open(Native Method)
     [java]     at java.io.FileInputStream.<init>(FileInputStream.java:120)
     [java]     at
org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:325)
     [java]     at org.apache.avro.Schema.parse(Schema.java:794)
     [java]     at
org.apache.gora.compiler.GoraCompiler.compileSchema(GoraCompiler.java:89)
     [java]     at
org.apache.gora.compiler.GoraCompiler.compileSchema(GoraCompiler.java:78)
     [java]     at
org.apache.gora.compiler.GoraCompiler.main(GoraCompiler.java:499)
     [java] Java Result: 1

It seems that the ':' file separator does not agree with the code. I
was therefore wondering how Avro's more recent (1.7.x)
SpecificCompiler reads in similar files, and how it deals with
multiple schemas.

Thanks very much in advance for any help.

Lewis

[0] http://svn.apache.org/repos/asf/gora/trunk/gora-core/src/main/java/org/apache/gora/compiler/GoraCompiler.java

--
Lewis
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB