Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume configuration fail-over problems


Copy link to this message
-
Re: Flume configuration fail-over problems
I changed my configuration to write to two HDFS instances to eliminate the
ElasticSearch sink.

I noticed the following exception on the source agent.

*2012-10-16 12:03:49,989 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.destroyConnection(AvroSink.java:199)] Avro
sink avroSink-1 closing avro client: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:03:49,993 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[WARN -
org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:185)]
Sink avroSink-1 failed and has been sent to failover list*
*org.apache.flume.EventDeliveryException: Failed to send events*
* at org.apache.flume.sink.AvroSink.process(AvroSink.java:325)*
* at
org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:182)
*
* at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
* at java.lang.Thread.run(Thread.java:662)*
*Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: localhost, port: 4545 }: Failed to send batch*
* at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:236)
*
* at org.apache.flume.sink.AvroSink.process(AvroSink.java:309)*
* ... 3 more*
*Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: localhost, port: 4545 }: Avro RPC call returned Status: FAILED*
* at
org.apache.flume.api.NettyAvroRpcClient.waitForStatusOK(NettyAvroRpcClient.java:312)
*
* at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:295)
*
* at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:224)
*
* ... 4 more*
*2012-10-16 12:03:51,998 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[INFO - org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:178)]
Avro sink avroSink-1: Building RpcClient with hostname: 127.0.0.1, port:
4545*
*2012-10-16 12:03:51,999 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
Batch size string = null*
*2012-10-16 12:03:52,001 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:192)] Avro
sink avroSink-1: Created RpcClient: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:03:55,008 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.destroyConnection(AvroSink.java:199)] Avro
sink avroSink-1 closing avro client: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:03:55,009 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.FailoverSinkProcessor$FailedSink.incFails(FailoverSinkProcessor.java:97)]
Sink avroSink-1 failed again, new refresh is at 1350414239009, current time
1350414235009*
*2012-10-16 12:03:59,341 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[INFO - org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:178)]
Avro sink avroSink-1: Building RpcClient with hostname: 127.0.0.1, port:
4545*
*2012-10-16 12:03:59,342 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
Batch size string = null*
*2012-10-16 12:03:59,347 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:192)] Avro
sink avroSink-1: Created RpcClient: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:04:13,849 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
Checking file:../conf/flume.conf for changes*
*2012-10-16 12:04:43,850 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
Checking file:../conf/flume.conf for changes*
*2012-10-16 12:05:13,850 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
Checking file:../conf/flume.conf for changes*

Then on the first collector

*2012-10-16 12:03:52,003 (New I/O server worker #1-2) [DEBUG -
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:244)] Avro
source avroSource-1: Received avro event batch of 100 events.*
*2012-10-16 12:03:55,004 (New I/O server worker #1-2) [ERROR -
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:261)] Avro
source avroSource-1: Unable to process event batch. Exception follows.*
*org.apache.flume.ChannelException: Unable to put batch on required
channel: org.apache.flume.channel.MemoryChannel{name: memoryChannel-1}*
* at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
*
* at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)*
* at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)*
* at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
*
* at java.lang.reflect.Method.invoke(Method.java:597)*
* at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
*
* at org.apache.avro.ipc.Responder.respond(Responder.java:149)*
* at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
*
* at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
*
* at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
*
* at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
*
* at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
*
* at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
*
* at
org.jboss.netty.handler.codec.frame.FrameDecoder