-
Avro sink errorRakos, Rudolf 2012-08-16, 12:34
Hi All,
We are developing a tool which embeds Flume. I have some questions about configuring Flume.
The Avro sink emits error level log messages every few seconds when it loses connection to the next hop Avro source.
Using a file channel before the Avro sink guarantees the transactional behavior and 100% delivery. So the messages will be delivered when the next hop Avro source comes online.
Is there a way to configure Flume to only log a warning level log message or only log this error once?
The log message:
2012-08-16 14:33:26,192 [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
at org.apache.flume.sink.AvroSink.process(AvroSink.java:325) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) ~[flume-ng-core-1.2.0.jar:1.2.0]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 11113 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:182) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:222) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.sink.AvroSink.process(AvroSink.java:282) ~[flume-ng-core-1.2.0.jar:1.2.0]
... 3 common frames omitted
Caused by: java.io.IOException: Error connecting to /127.0.0.1:11113
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) ~[avro-ipc-1.6.3.jar:1.6.3]
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198) ~[avro-ipc-1.6.3.jar:1.6.3]
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147) ~[avro-ipc-1.6.3.jar:1.6.3]
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
... 9 common frames omitted
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_31]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) ~[na:1.6.0_31]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401) ~[netty-3.2.7.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370) ~[netty-3.2.7.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292) ~[netty-3.2.7.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]
... 1 common frames omitted
Thank you for your help.
Regards,
Rudolf
Rudolf Rakos
Morgan Stanley | ISG Technology
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Be carbon conscious. Please consider our environment before printing this email.
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
We are developing a tool which embeds Flume. I have some questions about configuring Flume.
The Avro sink emits error level log messages every few seconds when it loses connection to the next hop Avro source.
Using a file channel before the Avro sink guarantees the transactional behavior and 100% delivery. So the messages will be delivered when the next hop Avro source comes online.
Is there a way to configure Flume to only log a warning level log message or only log this error once?
The log message:
2012-08-16 14:33:26,192 [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
at org.apache.flume.sink.AvroSink.process(AvroSink.java:325) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) ~[flume-ng-core-1.2.0.jar:1.2.0]
at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 11113 }: RPC connection error
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:182) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:222) ~[flume-ng-core-1.2.0.jar:1.2.0]
at org.apache.flume.sink.AvroSink.process(AvroSink.java:282) ~[flume-ng-core-1.2.0.jar:1.2.0]
... 3 common frames omitted
Caused by: java.io.IOException: Error connecting to /127.0.0.1:11113
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) ~[avro-ipc-1.6.3.jar:1.6.3]
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198) ~[avro-ipc-1.6.3.jar:1.6.3]
at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147) ~[avro-ipc-1.6.3.jar:1.6.3]
at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
... 9 common frames omitted
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_31]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) ~[na:1.6.0_31]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401) ~[netty-3.2.7.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370) ~[netty-3.2.7.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292) ~[netty-3.2.7.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]
... 1 common frames omitted
Thank you for your help.
Regards,
Rudolf
Rudolf Rakos
Morgan Stanley | ISG Technology
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Be carbon conscious. Please consider our environment before printing this email.
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
