Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # dev >> Python-Avro Codegen Proposal


+
Tal Levy 2013-06-03, 05:28
+
Stefan Krawczyk 2013-06-04, 01:28
+
Philip Zeyliger 2013-06-04, 18:24
+
Tal Levy 2013-06-04, 20:54
Copy link to this message
-
Re: Python-Avro Codegen Proposal
Thanks!  Please do file the JIRA.
https://cwiki.apache.org/AVRO/how-to-contribute.html is a guide on
contributing.

SpecificCompiler is template driven (or was, at some point).  I wonder if
you could have implemented it by just substituting a different set of
templates.

Cheers,

-- Philip
On Tue, Jun 4, 2013 at 1:54 PM, Tal Levy <[EMAIL PROTECTED]> wrote:

> I will try and clean it up and set up a patch to submit via JIRA
>
> here are the changes
>
> https://github.com/talevy/avro/tree/python-codegen
>
> a few caveats and thoughts about my current version:
>
> 1. I do not know how to best handle constructors, because some fields are
> not allowed to be null... maybe a builder pattern would work here, but it's
> kind of weird in python
> 2. I copy/pasted a lot of the code from SpecificCompiler to make the
> PythonCompiler... some renaming and code re-use via inheritance would make
> it read better.
> 3. I wanted to reuse the validate methods provided already in Avro to
> verify the record, but it takes away from some of the class type
> correctness for nested records and such.
> 4. I do not know what the best way of outputing multiple files is, I
> currently use the same packaging as the java classes into their namespace
> directories
> 5. I am not familiar with the avro-protocol format, so I only implemented
> enums and records.
>
> I updated the SpecificCompilerTool to have the following usage
>
> ```
> "Usage: [-string] (schema|protocol) (python|java) input... outputdir"
> ````
>
> So generating the python classes is as easy as java.
>
>
> let me know what you think
>
>
> On Tue, Jun 4, 2013 at 11:24 AM, Philip Zeyliger <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Tal,
> >
> > I would encourage you to file a JIRA and contribute your changes!
> >
> > I agree that the generated code approach is a great one for things like
> RPC
> > usage, where the schema changes slowly and the extra checking is super
> > handy.
> >
> > -- Philip
> >
> >
> > On Mon, Jun 3, 2013 at 6:28 PM, Stefan Krawczyk <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi Tal,
> > >
> > > I'm interested in using Avro + Python. That would sound useful, any
> > chance
> > > I could have a play with what you've done?
> > >
> > > Cheers,
> > >
> > > Stefan
> > >
> > >
> > > On Sun, Jun 2, 2013 at 10:28 PM, Tal Levy <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi,
> > > >
> > > > I recently started using Avro at my work and we found it difficult to
> > > keep
> > > > track of what python dict matched to what schema. Instead of having
> > > > random dicts being populated and then attempted to be serialized to
> > > avro, I
> > > > thought
> > > > it would be more readable and less error prone to codegen the python
> > dict
> > > > for developers. These classes are type checked field by field.
> Although
> > > it
> > > > does not
> > > > have the advantage of compiled type checking like in the java
> codegen,
> > it
> > > > is a
> > > > friendly wrapper around python dicts representing avro records to be
> > > > serialized.
> > > >
> > > > let me know what you think about this, I am still tweaking how it
> > > behaves.
> > > > I understand it is a bit unpythonic to enforce types in this way, but
> > the
> > > > readability
> > > > is worth it nonetheless.
> > > >
> > > > here is an example record:
> > > > https://gist.github.com/talevy/5696236
> > > >
> > > > I extended the avro compiler/tools to provide both java and python
> > > codegen
> > > > functionality.
> > > > so if this sounds like something others would use, maybe it makes
> sense
> > > to
> > > > include it
> > > > into the main repo.
> > > >
> > > >
> > > > Thanks,
> > > > Tal
> > > >
> > >
> >
>
+
Tal Levy 2013-06-05, 00:58