Hello,

I'm running into what looks like a limitation of the ant task for compiling
specific classes from schemas.   If this is documented somewhere, I
apologize, I couldn't find it.

I have three schemas in separate files, where two are referenced by the
third:

{
    "type": "record",
    "name": "MemberInfo",
    "fields": [
        {"name": "name", "type": "FullName"},
        {"name": "address", "type": "Address"},
    ]
}

{
    "type": "record",
    "name": "FullName",
    "fields": [
        {"name": "first", "type": "string"},
        {"name": "middle", "type": "string", "default": ""},
        {"name": "last", "type": "string"}
    ]
}

{
    "type": "record",
    "name": "Address",
    "fields": [
        {"name": "street", "type": "string"},
        {"name": "city", "type": "string"},
        {"name": "state", "type": "string"},
        {"name": "zip", "type": "int"}
    ]
}

When using the ant schema task, if I put these all in one file, only the
first is compiled.  If I supply three separate files, a separate Parser is
created for each one, so the reference from MemberInfo to Address/FullName
can't be resolved.

Of course it works to put the embedded schemas inline in a single
MemberInfo schema, but that's not the way we'd like to do it.

Rather than using the ant task I am going to try writing a small utility to
call this method in SpecificCompiler:

  public static void compileSchema(File[] srcFiles, File dest) throws
IOException

This method uses a single Parser instance for all input files, so it should
work.  Of course, I'll need to supply the files in the right order
(MemberInfo last).

Anything I'm missing?

Thanks,
--mark

On Fri, May 25, 2012 at 11:34 AM, Mark Hayes <[EMAIL PROTECTED]> wrote:
> I have three schemas in separate files, where two are referenced by the
> third:
> When using the ant schema task, if I put these all in one file, only the
> first is compiled.  If I supply three separate files, a separate Parser is
> created for each one, so the reference from MemberInfo to Address/FullName
> can't be resolved.
>

I found a way around this.  If I put all three schemas in a single JSON
array, in a single JSON document in a file, and parse that file using the
ant SchemaTask, then it works -- it generates classes for all three
schemas.  I assume it will also work when parsing the schema using other
tools.

I don't see this documented anywhere and I found it by trial and error.  My
guess is that the top level array is treated as an Avro union, and the
SpecificCompiler doesn't happen to output this top level union type.

I don't know if it is meant to work this way (i.e, will continue to work in
the future), or is just a coincidence.  Does anyone know?

Thanks,
--mark

On 05/30/2012 06:04 PM, Mark Hayes wrote:
> I found a way around this.  If I put all three schemas in a single JSON
> array, in a single JSON document in a file, and parse that file using
> the ant SchemaTask, then it works -- it generates classes for all three
> schemas.  I assume it will also work when parsing the schema using other
> tools.
>
> I don't see this documented anywhere and I found it by trial and error.
>   My guess is that the top level array is treated as an Avro union, and
> the SpecificCompiler doesn't happen to output this top level union type.

That's right.

> I don't know if it is meant to work this way (i.e, will continue to work
> in the future), or is just a coincidence.  Does anyone know?

This will continue to work in the future.  Other folks rely on it.

The Ant task might be improved to make this easier.  If the task reused
the SpecificCompiler instance then one might be able to specify the
compiler input as something like:

<fileset>
   <filename name="T1.avsc"/>
   <filename name="T2DependsonT1"/>
</fileset>

If this sounds useful to you, please file an issue in Jira.

Doug

On Thu, May 31, 2012 at 5:05 PM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> This will continue to work in the future.  Other folks rely on it.
>

Thank you Doug, it helps a lot to know that.
> The Ant task might be improved to make this easier.  If the task reused
> the SpecificCompiler instance then one might be able to specify the
> compiler input as something like:
>
> <fileset>
>  <filename name="T1.avsc"/>
>  <filename name="T2DependsonT1"/>
> </fileset>
>
> If this sounds useful to you, please file an issue in Jira.
It does seem useful, and also a natural enhancement, so I filed this Jira
issue:
https://issues.apache.org/jira/browse/AVRO-1107

Thanks again,
--mark