We've a client JVM process which uses flume client SDK (NettyAvroRPCClient)
to push events to a flume source which ultimately lands in HDFS.
On production we're still on flume 1.3, and one thing we find consistently
is that under heavy load, the client JVM hangs. We've narrowed it down to
the Flume client SDK,
>From what I suspect a long GC pause in flume agent, causes disconnects in
avro client, which can lead to client JVM hangs.
We're getting event's at the rate of about 25,000/sec, which are
distributed across 8 clients, and they in turn forward them to 24 flume
sources ( 6 boxes with 4 sources each). and each source writes to HDFS
(i.e. 24 HDFS sinks as well).
I tried switching flume agents GC to G1, which sort of helped, earlier the
client JVM hangs were about 5 mins apart, now it's about 10 mins, so there
is some progress.
Question is how to completely eliminate these hangs. The hang is so bad, I
can't even get the JVM to do a thread dump, so possible way for me to
investigate what caused the JVM to hang.
Could upgrading to 1.4, and using Thrift source help ?