Recently we switched over from Memory Channel to File Channel, as Memory
Channel has some GC issues.
Occasionally in File Channel I see this exception
org.apache.flume.ChannelException: Put queue for FileBackedTransaction of
capacity 5000 full, consider committing more frequently, increasing
capacity or increasing thread count. [channel=fileChannelD1]
Client batchSize is 1,000, and HDFS Sink batch size is also 1,000.
The channel capacity is 1M (1,000,000), and Channel Tx Capacity is 5,000
The underlying directories are not full, so the channel should have enough
space, nor does the channel has any backlog.
What I'm confused by are the 3 options the Exception mentions.
How do I , commit more frequently ? or increase capacity ? (Capacity of
Channel is 1M, and that is not full), or increase thread count ?( I see no
option of thread count in file channel, or is this referring to threadcout
of the HDFS sink which reads from this sink ?)
Lastly, would GC in Hadoop (mostly Namenode) cause HDFS Timeout issues in
HDFS Sink, coz we see HDFS Timeout errors, more or less at the same time
across all our flume nodes, so I suspect it could be NameNode GC causing