Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Timeouts when using avro sink/source


Copy link to this message
-
Re: Timeouts when using avro sink/source
BTW both machines are AWS servers in same region (US East)
On Mon, Sep 30, 2013 at 8:42 AM, Anat Rozenzon <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I'm trying to send events from flume agent to flume central collector
> using avro sink/source pair.
>
> This is the avro sink configuration, I have two sinks to drain the channel:
>
> agent.sinks.AvroSink1-1.type = avro
> agent.sinks.AvroSink1-1.channel = fileChannel
> agent.sinks.AvroSink1-1.hostname = X.X.X.X
> agent.sinks.AvroSink1-1.port = 45451
> agent.sinks.AvroSink1-1.connect-timeout = 60000
> agent.sinks.AvroSink1-1.request-timeout = 60000
> agent.sinks.AvroSink1-1.batch-size = 1000
> agent.sinks.AvroSink1-1.compression-type=deflate
> agent.sinks.AvroSink1-1.compression-level=9
>
> agent.sinks.AvroSink1-2.type = avro
> agent.sinks.AvroSink1-2.channel = fileChannel
> agent.sinks.AvroSink1-2.hostname = X.X.X.X
> agent.sinks.AvroSink1-2.port = 45451
> agent.sinks.AvroSink1-2.connect-timeout = 60000
> agent.sinks.AvroSink1-2.request-timeout = 60000
> agent.sinks.AvroSink1-2.batch-size = 1000
> agent.sinks.AvroSink1-2.compression-type=deflate
> agent.sinks.AvroSink1-2.compression-level=9
>
>
> However I see many connectivity timeout exceptions although when I try to
> connect directly using telnet there is no problem.
> Anything I can change or check to improve this?
>
> 30 Sep 2013 01:32:56,600 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to send events
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: X.X.X.X, port: 45451 }: Failed to send batch
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:294)
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:366)
>         ... 3 more
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: X.X.X.X: 45451 }: Handshake timed out after 60000ms
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:338)
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:282)
>         ... 4 more
> Caused by: java.util.concurrent.TimeoutException
>         at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:258)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:119)
>         at
> org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:336)
>         ... 5 more
> 30 Sep 2013 01:32:57,212 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.AbstractRpcSink.createConnection:205)  - Rpc sink
> AvroSink1-1: Building RpcClient with hostname: X.X.X.X, port: 45451
> 30 Sep 2013 01:32:57,213 INFO
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.sink.AvroSink.initializeRpcClient:126)  - Attempting to
> create Avro Rpc client.
> 30 Sep 2013 01:32:58,136 ERROR
> [SinkRunner-PollingRunner-DefaultSinkProcessor]
> (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver
> event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to send events
>         at
> org.apache.flume.sink.AbstractRpcSink.process(AbstractRpcSink.java:382)
>         at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>         at
> org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>         at java.lang.Thread.run(Thread.java:724)
> Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
> host: X.X.X.X, port: 45451 }: Failed to send batch
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB