Flume, mail # user - Re: Maximum size of the request accepted by Avro Source - 2014-04-25, 12:48
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
Re: Maximum size of the request accepted by Avro Source
This is bound to happen for large file. Avro's Netty implementation uses
FrameDecoder, which keep the chunks in memory till the data is complete for
decoding. So for larger files OOME shall happen sooner or later. The max
size is more related to memory available than a fixed number. More number
of concurrent connection uploading would result in OOME faster.

The solution is to use a decoder that can push data to file system, rather
than keeping in memory.

There shall be similar implementation or you may need to write. This change
needs to be done in Avro code base, check org.apache.avro.ipc.NettyServer
Line#97 (for Avro 1.7.3)

On Tue, Apr 22, 2014 at 2:59 PM, Himanshu Patidar <

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB