-Re: 3x faster python reader
Philip Zeyliger 2013-04-29, 17:56
Once you post to the JIRA, I'd be happy to review it.
On Mon, Apr 29, 2013 at 9:22 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> This sounds awesome! Is the API compatible with the existing API? If
> it's incompatible and cannot easily be made compatible then perhaps we
> can add it as the 'new' API and deprecate the old one. Regardless,
> please file an issue in Jira (issues.apache.org/jira/browse/AVRO) and
> attach your patch there.
> On Sun, Apr 28, 2013 at 10:24 PM, Uri Laserson <[EMAIL PROTECTED]>
> > Hi all,
> > I rewrote some of the python code to read avro files. I was able to
> > achieve a ~3x speedup over the current impl, and can probably do better
> > it was cleaned up more. The main changes are:
> > * Eliminated the object-oriented nature of the reader. It's just
> > now. Presumably this can be changed back, but it didn't really seem like
> > there was any reason for it.
> > * Given a reader and writer schema, it precomputes as much helpful info
> > it can upfront and caches this in a dictionary that the read functions
> > * The code is compiled with Cython for speedup.
> > How can this be used to improve the current python api? Let me know how
> > can be helpful...
> > Uri
> > --
> > Uri Laserson, PhD
> > Data Scientist, Cloudera
> > Twitter/GitHub: @laserson
> > +1 617 910 0447
> > [EMAIL PROTECTED]