Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Avro sink error

Copy link to this message
Re: Avro sink error
Hari Shreedharan 2012-08-16, 18:08
Hi Rudolf,

Unfortunately, the sink will try to connect to the source and when it fails it will log this error message. The sink would back off for a few seconds(it increases with each failure till a max of 5 seconds I think), but each time it tries to connect it will log this error. Please file a jira if you want this to be reduced to a WARN, so it can be discussed with other devs and if a consensus is reached, the current code can be patched to make this a warn.


Hari Shreedharan
On Thursday, August 16, 2012 at 5:34 AM, Rakos, Rudolf wrote:

> Hi All,
> We are developing a tool which embeds Flume. I have some questions about configuring Flume.
> The Avro sink emits error level log messages every few seconds when it loses connection to the next hop Avro source.
> Using a file channel before the Avro sink guarantees the transactional behavior and 100% delivery. So the messages will be delivered when the next hop Avro source comes online.
> Is there a way to configure Flume to only log a warning level log message or only log this error once?
> The log message:
> 2012-08-16 14:33:26,192 [SinkRunner-PollingRunner-DefaultSinkProcessor] ERROR org.apache.flume.SinkRunner - Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: Failed to send events
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:325) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31]
> Caused by: org.apache.flume.FlumeException: NettyAvroRpcClient { host: localhost, port: 11113 }: RPC connection error
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.api.RpcClientFactory.getInstance(RpcClientFactory.java:88) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:182) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.AvroSink.verifyConnection(AvroSink.java:222) ~[flume-ng-core-1.2.0.jar:1.2.0]
> at org.apache.flume.sink.AvroSink.process(AvroSink.java:282) ~[flume-ng-core-1.2.0.jar:1.2.0]
> ... 3 common frames omitted
> Caused by: java.io.IOException: Error connecting to /
> at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:249) ~[avro-ipc-1.6.3.jar:1.6.3]
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:198) ~[avro-ipc-1.6.3.jar:1.6.3]
> at org.apache.avro.ipc.NettyTransceiver.<init>(NettyTransceiver.java:147) ~[avro-ipc-1.6.3.jar:1.6.3]
> at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106) ~[flume-ng-sdk-1.2.0.jar:1.2.0]
> ... 9 common frames omitted
> Caused by: java.net.ConnectException: Connection refused: no further information
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_31]
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) ~[na:1.6.0_31]
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:401) ~[netty-3.2.7.Final.jar:na]
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:370) ~[netty-3.2.7.Final.jar:na]
> at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:292) ~[netty-3.2.7.Final.jar:na]
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]