Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Speed improvements for reflection-based serialization

Leo Romanoff 2013-03-21, 07:39
Doug Cutting 2013-03-21, 16:30
Copy link to this message
Re: Speed improvements for reflection-based serialization
More generally, Avro contribution guidelines are at:



On Thu, Mar 21, 2013 at 9:30 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:
> Leo,
> Please submit your patches, I'd love to see them.  Create an issue in
> Jira and attach your changes there.
> https://issues.apache.org/jira/browse/AVRO
> Thanks,
> Doug
> On Thu, Mar 21, 2013 at 12:39 AM, Leo Romanoff <[EMAIL PROTECTED]> wrote:
>> Hi,
>> I've played a bit with Avro serialization based on reflection, i.e. with the classes from org.apache.avro.generic package.
>> It works fine in this mode, but is rather slow compared to such frameworks like protostuff or kryo.
>> Quick look at the source code has shown that a lot of reflection-based operations and class lookups are not cached. And such operations are usually pretty expensive when executed by a JVM.
>> So I changed some of org.apache.avro.generic classes, introduced caching and a few other optimizations. Now it seems to perform much better.
>> I could submit my patch for a review, if anyone is interested in such improvements. I'm new to Avro, but I got the impression that Voldermont and may be a few other BigData projects are using it? May be they are interested, though I don't know if they use this reflection-based serialization.
>> Best Regards,
>>   Leo