Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Avro sink error


Copy link to this message
-
Avro sink error
Hi All,

We are developing a tool which embeds Flume. I have some questions about configuring Flume.

The Avro sink emits error level log messages every few seconds when it loses connection to the next hop Avro source.
Using a file channel before the Avro sink guarantees the transactional behavior and 100% delivery. So the messages will be delivered when the next hop Avro source comes online.
Is there a way to configure Flume to only log a warning level log message or only log this error once?

The log message:
2012-08-16 14:33:26,192 [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: Failed to send events
                at org.apache.flume.sink.AvroSink.process(AvroSink.java:325) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 11113 }: RPC connection error
                at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:182) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:222) ~[flume-ng-core-1.2.0.jar:1.2.0]
                at org.apache.flume.sink.AvroSink.process(AvroSink.java:282) ~[flume-ng-core-1.2.0.jar:1.2.0]
                ... 3 common frames omitted
Caused by: java.io.IOException: Error connecting to /127.0.0.1:11113
                at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) ~[avro-ipc-1.6.3.jar:1.6.3]
                at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198) ~[avro-ipc-1.6.3.jar:1.6.3]
                at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147) ~[avro-ipc-1.6.3.jar:1.6.3]
                at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
                ... 9 common frames omitted
Caused by: java.net.ConnectException: Connection refused: no further information
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_31]
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) ~[na:1.6.0_31]
                at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401) ~[netty-3.2.7.Final.jar:na]
                at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370) ~[netty-3.2.7.Final.jar:na]
                at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292) ~[netty-3.2.7.Final.jar:na]
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]
                ... 1 common frames omitted

Thank you for your help.

Regards,
Rudolf

Rudolf Rakos
Morgan Stanley | ISG Technology
[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
Be carbon conscious. Please consider our environment before printing this email.
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB