Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - flume.EventDeliveryException: Failed to send events


Copy link to this message
-
Re: flume.EventDeliveryException: Failed to send events
Brock Noland 2013-04-16, 19:26
Another possibility is that the downstream agent is near capacity from a
memory perspective. What is your heap size for these agents?
On Tue, Apr 16, 2013 at 2:19 PM, Hari Shreedharan <[EMAIL PROTECTED]
> wrote:

>  One possibility is that you are hitting the file channel disk too hard -
> do you have just one disk for checkpoint and data? It might be getting slow
> because of this? Also you should probably just remove the thread limit on
> AvroSource. It usually does not cause too much havoc unless you have a
> massive number of connections causing too many threads.
>
> --
> Hari Shreedharan
>
> On Tuesday, April 16, 2013 at 12:07 PM, Chris Neal wrote:
>
> Thanks Hari.
>
> I increased both the connect and request timeouts to 40000ms, and I'm
> testing that now.  I am talking on a LAN though, which is part of the
> reason I'm concerned.  Seems like it might not actually be a network issue,
> but perhaps an overloaded AvroSource on the back end?
>
>
> On Tue, Apr 16, 2013 at 1:52 PM, Hari Shreedharan <
> [EMAIL PROTECTED]> wrote:
>
>  Looks like you are hitting Avro IPC timeouts - you should probably
> increase it, especially if you are talking over WAN.
>
> --
> Hari Shreedharan
>
> On Tuesday, April 16, 2013 at 11:38 AM, Chris Neal wrote:
>
> I'm seeing the same thing :)
>
> Mine is all on a local LAN though, so the fact that an RPC call doesn't
> reply in 10000ms or 20000ms is quite odd.  My configuration is for the most
> part the same as Denis' configuration.  Two tiered system, ExecSources
> running tail -F on log files to an AvroSink, to an AvroSource, loading into
> HDFS on the back tier.
>
> I, too, see this on the AvroSink
>
> Either (A):
> [2013-04-15 23:57:14.827]
> [org.apache.flume.sink.LoadBalancingSinkProcessor] [ WARN]
> [SinkRunner-PollingRunner-LoadBalancingSinkProcessor] []
>  (LoadBalancingSinkProcessor.java:process:154) Sink failed to consume
> event. Attempting next sink if available.
> org.apache.flume.EventDeliveryException: Failed to send events
>         at org.apache.flume.sink.AvroSink.process(AvroSink.java:324)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:151)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: hadoopjt01.pegs.com, port: 10000 }: Failed to send batch
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:236)
>         at org.apache.flume.sink.AvroSink.process(AvroSink.java:308)
>         ... 3 more
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: hadoopjt01.pegs.com, port: 10000 }: Handshake timed out after
> 20000ms
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:280)
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:224)
>         ... 4 more
> Caused by: java.util.concurrent.TimeoutException
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:228)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:91)
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:278)
>         ... 5 more
>
> or (B):
> [2013-04-15 19:49:01.135]
> [org.apache.flume.sink.LoadBalancingSinkProcessor] [ WARN]
> [SinkRunner-PollingRunner-LoadBalancingSinkProcessor] []
>  (LoadBalancingSinkProcessor.java:process:154) Sink failed to consume
> event. Attempting next sink if available.
> org.apache.flume.EventDeliveryException: Failed to send events
>         at org.apache.flume.sink.AvroSink.process(AvroSink.java:324)
>         at
> org.apache.flume.sink.LoadBalancingSinkProcessor.process(LoadBalancingSinkProcessor.java:151)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:619)
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org