Hi All,

We are developing a tool which embeds Flume. I have some questions about configuring Flume.

The Avro sink emits error level log messages every few seconds when it loses connection to the next hop Avro source.
Using a file channel before the Avro sink guarantees the transactional behavior and 100% delivery. So the messages will be delivered when the next hop Avro source comes online.
Is there a way to configure Flume to only log a warning level log message or only log this error once?

The log message:
2012-08-16 14:33:26,192 [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
                at org.apache.flume.sink.AvroSink.process(AvroSink.java:325) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 11113 }: RPC connection error
                at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:182) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:222) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.AvroSink.process(AvroSink.java:282) ~[flume-ng-core-1.2.0.jar:1.2.0]
                ... 3 common frames omitted
Caused by: java.io.IOException: Error connecting to /127.0.0.1:11113
                at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) ~[avro-ipc-1.6.3.jar:1.6.3]
                at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198) ~[avro-ipc-1.6.3.jar:1.6.3]
                at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147) ~[avro-ipc-1.6.3.jar:1.6.3]
                at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                ... 9 common frames omitted
Caused by: java.net.ConnectException: Connection refused: no further information
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_31]
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) ~[na:1.6.0_31]
                at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401) ~[netty-3.2.7.Final.jar:na]
                at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370) ~[netty-3.2.7.Final.jar:na]
                at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292) ~[netty-3.2.7.Final.jar:na]
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]
                ... 1 common frames omitted

Thank you for your help.

Regards,
Rudolf

Rudolf Rakos
Morgan Stanley | ISG Technology
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Be carbon conscious. Please consider our environment before printing this email.
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

Hi Rudolf,

Unfortunately, the sink will try to connect to the source and when it fails it will log this error message. The sink would back off for a few seconds(it increases with each failure till a max of 5 seconds I think), but each time it tries to connect it will log this error. Please file a jira if you want this to be reduced to a WARN, so it can be discussed with other devs and if a consensus is reached, the current code can be patched to make this a warn.

Thanks,
Hari

--
Hari Shreedharan
On Thursday, August 16, 2012 at 5:34 AM, Rakos, Rudolf wrote:

> Hi All,
>
> We are developing a tool which embeds Flume. I have some questions about configuring Flume.
>
> The Avro sink emits error level log messages every few seconds when it loses connection to the next hop Avro source.
> Using a file channel before the Avro sink guarantees the transactional behavior and 100% delivery. So the messages will be delivered when the next hop Avro source comes online.
> Is there a way to configure Flume to only log a warning level log message or only log this error once?
>
> The log message:
> 2012-08-16 14:33:26,192 [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to send events
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:325) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
> Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 11113 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:182) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:222) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:282) ~[flume-ng-core-1.2.0.jar:1.2.0]
> ... 3 common frames omitted
> Caused by: java.io.IOException: Error connecting to /127.0.0.1:11113
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) ~[avro-ipc-1.6.3.jar:1.6.3]
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198) ~[avro-ipc-1.6.3.jar:1.6.3]
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147) ~[avro-ipc-1.6.3.jar:1.6.3]
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> ... 9 common frames omitted
> Caused by: java.net.ConnectException: Connection refused: no further information
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_31]
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) ~[na:1.6.0_31]
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401) ~[netty-3.2.7.Final.jar:na]
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370) ~[netty-3.2.7.Final.jar:na]
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292) ~[netty-3.2.7.Final.jar:na]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]