Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - avro object reuse


Copy link to this message
-
RE: avro object reuse
ey-chih chow 2011-06-09, 22:26

In addition, we ran the same MR job once again and got the following histogram.  Whey this is different from the previous one?  Thanks.
Ey-Chih Chow

Object Histogram:

num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              4327    100242096       byte[]
2:              2050    5381496 int[]
3:              23762   2822864 * ConstMethodKlass
4:              23762   1904760 * MethodKlass
5:              39295   1688992 * SymbolKlass
6:              2127    1216976 * ConstantPoolKlass
7:              2127    882760  * InstanceKlassKlass
8:              11298   773008  char[]
9:              1847    742936  * ConstantPoolCacheKlass
10:             1064    297448  * MethodDataKlass
11:             11387   273288  java.lang.String
12:             2317    222432  java.lang.Class
13:             3288    204440  short[]
14:             3167    156664  * System ObjArray
15:             1360    86720   java.util.HashMap$Entry[]
16:             535     85600   org.codehaus.jackson.impl.ReaderBasedParser
17:             3498    83952   java.util.HashMap$Entry
18:             666     53280   java.lang.reflect.Method
19:             161     52808   * ObjArrayKlassKlass
20:             1267    44704   java.lang.Object[]
21:             1808    43392   java.util.Hashtable$Entry
22:             1070    42800   org.codehaus.jackson.impl.JsonReadContext
23:             777     31080   java.util.HashMap
24:             535     29960   org.codehaus.jackson.util.TextBuffer
25:             567     27216   java.nio.HeapByteBuffer
26:             553     26544   org.apache.avro.Schema$Props
27:             549     26352   java.nio.HeapCharBuffer
28:             538     25824   org.codehaus.jackson.map.DeserializationConfig
29:             535     25680   org.codehaus.jackson.io.IOContext
30:             1554    24864   org.codehaus.jackson.sym.CharsToNameCanonicalizer$Bucket
31:             539     21560   org.codehaus.jackson.sym.CharsToNameCanonicalizer
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: RE: avro object reuse
Date: Thu, 9 Jun 2011 15:16:29 -0700
I forgot to mention that the histogram in my previous message was extracted from a mapper of one of our MR job.
Ey-Chih Chow

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: RE: avro object reuse
Date: Thu, 9 Jun 2011 15:08:02 -0700
We did more monitoring.  At one instance, we got the following histogram via Jmap.  The question is why there are so many instances of BinaryDecoder$BufferAccessor and BinaryDecoder$ByteArrayByteSource.  How to avoid this?  Thanks.

Object Histogram:

num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              4199    100241168       byte[]
2:              272948  8734336 org.apache.avro.io.BinaryDecoder$BufferAccessor
3:              272945  8734240 org.apache.avro.io.BinaryDecoder$ByteArrayByteSource
4:              2093    5387976 int[]
5:              23762   2822864 * ConstMethodKlass
6:              23762   1904760 * MethodKlass
7:              39295   1688992 * SymbolKlass
8:              2127    1216976 * ConstantPoolKlass
9:              2127    882760  * InstanceKlassKlass
10:             1847    742936  * ConstantPoolCacheKlass
11:             9602    715608  char[]
12:             1072    299584  * MethodDataKlass
13:             9698    232752  java.lang.String
14:             2317    222432  java.lang.Class
15:             3288    204440  short[]
16:             3167    156664  * System ObjArray
17:             2401    57624   java.util.HashMap$Entry
18:             666     53280   java.lang.reflect.Method
19:             161     52808   * ObjArrayKlassKlass
20:             1808    43392   java.util.Hashtable$Entry
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: RE: avro object reuse
Date: Wed, 1 Jun 2011 15:14:03 -0700
We use a lot of toString() call on the avro Utf8 object.  Will this cause Jackson call?  Thanks.
Ey-Chih

From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Date: Wed, 1 Jun 2011 13:38:39 -0700
Subject: Re: avro object reuse
This is great info.
Jackson should only be used once when the file is opened, so this is confusing from that point of view.  Is something else using Jackson or initializing an Avro JsonDecoder frequently?  There are over 100000 Jackson DeserializationConfig objects.
Another place that parses the schema is in AvroSerialization.java.  Does the Hadoop getDeserializer() API method get called once per job, or per record?  If this is called more than once per map job, it might explain this.
In principle, Jackson is only used by a mapper during initialization.  The below indicates that this may not be the case or that something outside of Avro is causing a lot of Jackson JSON parsing.
Are you using something that is converting the Avro data to Json form?  toString() on most Avro datum objects will do a lot of work with Jackson, for example — but the below are deserializer objects not serializer objects so that is not likely the issue.
On 6/1/11 11:34 AM, "ey-chih chow" <[EMAIL PROTECTED]> wrote:

We ran jmap on one of our mapper and found the top usage as follows:
num  #instances #bytes Class description--------------------------------------------------------------------------1: 24405 291733256 byte[]2: 6056 40228984 int[]3: 388799 19966776 char[]4: 101779 16284640 org.codehaus.jackson.impl.ReaderBasedParser5: 369623 11827936 java.lang.String6: 111059 8769424 java.util.HashMap$Entry[]7: 204083 8163320 org.codehaus.jackson.impl.JsonReadContext8: 211374 6763968 java.util.HashMap$Entry9: 102551 5742856 org.codehaus.jackson.util.TextBuffer10: 105854 5080992 java.nio.HeapByteBuffer11: 105821 5079408 java.nio.HeapCharBuffer12: 104578 5019744 java.util.HashMap13: 102551 4922448 org.codehaus.jackson.io.IOContext14: 101782 4885536 org.code