Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> NettyAvroRPCClient hangs Client JVM


Copy link to this message
-
Re: NettyAvroRPCClient hangs Client JVM
Hi Hari,

Yes I am using memory channel. The boxes have 16GB RAMs, we're running with
8GB heap each.
Each memory channel capacity is 1 Million (4 Sinks so 4 Million in total),
and transaction size is 10K per sink.

Batch size is also set to 10K, we've played with these values , but the
issue is persistent.
Can you elaborate as to what issues these are, and what exactly takes place
?
A step by step deconstruction of the problem would really help me in
understanding what's going on.

thanks
Bhaskar
On Fri, Oct 4, 2013 at 11:57 AM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  Are you using the Memory Channel on the agents? We do know that there
> might be some issues when the memory channel is used when the heap is
> pretty large. We are wokring to resolve it.
>
>
> Thanks,
> Hari
>
> On Friday, October 4, 2013 at 5:34 AM, Bhaskar V. Karambelkar wrote:
>
> We've a client JVM process which uses flume client SDK
> (NettyAvroRPCClient) to push events to a flume source which ultimately
> lands in HDFS.
>
> On production we're still on flume 1.3, and one thing we find consistently
> is that under heavy load, the client JVM hangs. We've narrowed it down to
> the Flume client SDK,
>
> From what I suspect a long GC pause in flume agent, causes disconnects in
> avro client, which can lead to client JVM hangs.
>
> We're getting event's at the rate of about 25,000/sec, which are
> distributed across 8 clients, and they in turn forward them to 24 flume
> sources ( 6 boxes with 4 sources each). and each source writes to HDFS
> (i.e. 24 HDFS sinks as well).
>
> I tried switching flume agents GC to G1, which sort of helped, earlier the
> client JVM hangs were about 5 mins apart, now it's about 10 mins, so there
> is some progress.
>
> Question is how to completely eliminate these hangs. The hang is so bad, I
> can't even get the JVM to do a thread dump, so possible way for me to
> investigate what caused the JVM to hang.
>
> Could upgrading to 1.4, and using Thrift source help ?
>
>
>