Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> avro object reuse


Copy link to this message
-
RE: avro object reuse

I actually looked into Avro code to find out how Avro does object reuse.  I looked at AvroUtf8InputFormat and got the following question.  Why a new Utf8 object has to be created each time the method next(AvroWrapper<Utf8> key, NullWritable value) is called ?  Will this eat up too much memory when we call next(key, value) many times?  Since Utf8 is mutable, can we just create one Utf8 object for all the calls to next(key, value)?  Will this save memory?  Thanks.
Ey-Chih Chow

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: avro object reuse
Date: Tue, 31 May 2011 10:38:39 -0700
Hi,
We have several mapreduce jobs using avro.  They take too much memory when running on production.  Can anybody suggest some object reuse techniques to cut down memory usage?  Thanks.
Ey-Chih Chow    
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB