Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # dev >> Avro C# enhancements


Copy link to this message
-
Avro C# enhancements
After using Avro successfully for a couple of years now around the generic model we recently integrated the specific model into our codebase.  In order for this to work we needed to make some enhancements to the Avro core in order to allow Avro to integrate into our existing type system.  Just wondering if there would be any interest in the following features for the C# code base.

1. Avro.Specific.ObjectCreator decoupling.  Currently there is a singleton instance of this class.  We needed to override the default behaviour for name to type mapping and object creation in order to integrate with our type system.  In some more esoteric cases we required a specific ObjectCreator for a given reader/writer.  What we have done is to introduce a very simple IObjectCreator interface with two methods, GetType and GetConstructor.  Each reader/writer can optionally have its own instance of IObjectCreator with the default behaviour to use the global singleton instance, i.e. current behaviour.  In addition to the new interface we made many of the methods on ObjectCreator protected and split up some larger methods in order allow for implementation reuse.

2. Block writers.  The current generic writers only work with arrays, and the current specific writers only work with IList.  This was problematic as in some cases we needed to translate objects to be serialized essentially requiring a duplicate copy of the objects to be held in memory (at least for the array case).  As our environment was already memory constrained this extra copy would be too great in some cases.  What we implemented was support for writing arrays in blocks from an IEnumerable/IEnumerable<>.  This allows for serializing a Linq translation pipeline for example.  These writers (4 in total as we implemented for both generic/specific and original/datum versions of the writers) are all new classes and only required minimal changes to the existing classes, largely just making some of the existing implementation protected so we could inherit.

3. CodeGen inheritance.  We reworked the CodeGen class so most of the implementation was protected and methods were finer grained.  We also introduced more options for controlling the generated names.  As integration with our existing type system required non-standard generated classes we needed a custom code generator.  The changes to CodeGen allowed us to reuse most of the existing implementation.

4. HTTP transceiver.  Not related to our custom type system but needed to connect to a Java based Avro service.

If there is any interest in these changes I will put together some patches and tests as time permits.  The HTTP change is fairly standalone.  Items 1 and 2 are a little more complicated to disentangle as they touch common code.

Dave
This e-mail and any attachments may contain confidential and privileged information. If you are not the intended recipient, please notify the sender immediately by return e-mail, delete this e-mail and destroy any copies. Any dissemination or use of this information by a person other than the intended recipient is unauthorized and may be illegal. Unless otherwise stated, opinions expressed in this e-mail are those of the author and are not endorsed by the author's employer.

Le présent message, ainsi que tout fichier qui y est joint, est envoyé à l'intention exclusive de son ou de ses destinataires; il est de nature confidentielle et peut constituer une information privilégiée. Nous avertissons toute personne autre que le destinataire prévu que tout examen, réacheminement, impression, copie, distribution ou autre utilisation de ce message et de tout fichier qui y est joint est strictement interdit. Si vous n'êtes pas le destinataire prévu, veuillez en aviser immédiatement l'expéditeur par retour de courriel et supprimer ce message et tout  document joint de votre système. Sauf indication contraire, les opinions exprimées dans le présent message sont celles de l’auteur et ne sont pas avalisées par l’employeur de l’auteur.