Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Timeouts when using avro sink/source


Copy link to this message
-
Re: Timeouts when using avro sink/source
BTW both machines are AWS servers in same region (US East)
On Mon, Sep 30, 2013 at 8:42 AM, Anat Rozenzon <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm trying to send events from flume agent to flume central collector
> using avro sink/source pair.
>
> This is the avro sink configuration, I have two sinks to drain the channel:
>
> agent.sinks.AvroSink1-1.type = avro
> agent.sinks.AvroSink1-1.channel = fileChannel
> agent.sinks.AvroSink1-1.hostname = X.X.X.X
> agent.sinks.AvroSink1-1.port = 45451
> agent.sinks.AvroSink1-1.connect-timeout = 60000
> agent.sinks.AvroSink1-1.request-timeout = 60000
> agent.sinks.AvroSink1-1.batch-size = 1000
> agent.sinks.AvroSink1-1.compression-type=deflate
> agent.sinks.AvroSink1-1.compression-level=9
>
> agent.sinks.AvroSink1-2.type = avro
> agent.sinks.AvroSink1-2.channel = fileChannel
> agent.sinks.AvroSink1-2.hostname = X.X.X.X
> agent.sinks.AvroSink1-2.port = 45451
> agent.sinks.AvroSink1-2.connect-timeout = 60000
> agent.sinks.AvroSink1-2.request-timeout = 60000
> agent.sinks.AvroSink1-2.batch-size = 1000
> agent.sinks.AvroSink1-2.compression-type=deflate
> agent.sinks.AvroSink1-2.compression-level=9
>
>
> However I see many connectivity timeout exceptions although when I try to
> connect directly using telnet there is no problem.
> Anything I can change or check to improve this?
>
> 30 Sep 2013 01:32:56,600 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to send events
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: X.X.X.X, port: 45451 }: Failed to send batch
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:294)
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:366)
>         ... 3 more
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: X.X.X.X: 45451 }: Handshake timed out after 60000ms
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:338)
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:282)
>         ... 4 more
> Caused by: java.util.concurrent.TimeoutException
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:336)
>         ... 5 more
> 30 Sep 2013 01:32:57,212 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.AbstractRpcSink.createConnection:205)  - Rpc sink
> AvroSink1-1: Building RpcClient with hostname: X.X.X.X, port: 45451
> 30 Sep 2013 01:32:57,213 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.AvroSink.initializeRpcClient:126)  - Attempting to
> create Avro Rpc client.
> 30 Sep 2013 01:32:58,136 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to send events
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: X.X.X.X, port: 45451 }: Failed to send batch