-ExecSource->MemoryChannel->AvroSink->AvroSource->FileChannel->HDFSSink throughput question
I need some thoughts on sizing/tuning of the above (common) route in
FlumeNG to maximize throughput. Here is my setup:
*Source JVM (ExecSource/MemoryChannel/AvroSink):*
Number of ExecSources in config: 124 (yes, it's a ton. Can't do anything
about it :) The write rate to the source files is fairly fast and bursty.
ExecSource.batchSize = 1000
(so, when all 124 tail -F instances get 1000 events, they all dump to the
MemoryChannel.capacity = 1000000
MemoryChannel.transactionCapacity = 1000
(somewhat unclear on what this is. Docs say "The number of events stored
in the channel per transaction", but what is a "transaction" to a
AvroSink.batchSize = 1000
*Destination JVM (AvroSource/FileChannel/HDFSSink)*
(Cluster of two JVMs on two servers, each configured the same as per below)
-XX:MaxDirectMemorySize is not defined, so whatever the default is
AvroSource.threads = 64
FileChannel.transactionCapacity = 1000
FileChannel.capacity = 32000000
HDFSSink.batchSize = 1000
HDFSSink.threadPoolSize = 64
With this configuration, in about 5 minutes, I get the common Exception:
"Space for commit to queue couldn't be acquired Sinks are likely not
keeping up with sources, or the buffer size is too tight"
on the Source JVM. It is no where near the 4g max, rather only at about
I'm wondering about the logic of having all the batch sizes/transaction
sizes 1000. My thought was that would keep from fragmenting the transfer
of data, but maybe that's flawed? Should the sizes be different?
Also curious about increasing the MaxDirectMemorySize to something larger
than 256MB? I tried removing it altogether in my Source JVM (which makes
the size unbounded), but that didn't seem to make a difference.
I'm having some trouble figuring out where the backup is happening, and how
to open up the gates. :)
Thanks in advance for any suggestions.