Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Flume configuration fail-over problems


Copy link to this message
-
Re: Flume configuration fail-over problems
I changed my configuration to write to two HDFS instances to eliminate the
ElasticSearch sink.

I noticed the following exception on the source agent.

*2012-10-16 12:03:49,989 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.destroyConnection(AvroSink.java:199)] Avro
sink avroSink-1 closing avro client: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:03:49,993 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[WARN -
org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:185)]
Sink avroSink-1 failed and has been sent to failover list*
*org.apache.flume.EventDeliveryException: Failed to send events*
* at org.apache.flume.sink.AvroSink.process(AvroSink.java:325)*
* at
org.apache.flume.sink.FailoverSinkProcessor.process(FailoverSinkProcessor.java:182)
*
* at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)*
* at java.lang.Thread.run(Thread.java:662)*
*Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: localhost, port: 4545 }: Failed to send batch*
* at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:236)
*
* at org.apache.flume.sink.AvroSink.process(AvroSink.java:309)*
* ... 3 more*
*Caused by: org.apache.flume.EventDeliveryException: NettyAvroRpcClient {
host: localhost, port: 4545 }: Avro RPC call returned Status: FAILED*
* at
org.apache.flume.api.NettyAvroRpcClient.waitForStatusOK(NettyAvroRpcClient.java:312)
*
* at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:295)
*
* at
org.apache.flume.api.NettyAvroRpcClient.appendBatch(NettyAvroRpcClient.java:224)
*
* ... 4 more*
*2012-10-16 12:03:51,998 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[INFO - org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:178)]
Avro sink avroSink-1: Building RpcClient with hostname: 127.0.0.1, port:
4545*
*2012-10-16 12:03:51,999 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
Batch size string = null*
*2012-10-16 12:03:52,001 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:192)] Avro
sink avroSink-1: Created RpcClient: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:03:55,008 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.destroyConnection(AvroSink.java:199)] Avro
sink avroSink-1 closing avro client: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:03:55,009 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.FailoverSinkProcessor$FailedSink.incFails(FailoverSinkProcessor.java:97)]
Sink avroSink-1 failed again, new refresh is at 1350414239009, current time
1350414235009*
*2012-10-16 12:03:59,341 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[INFO - org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:178)]
Avro sink avroSink-1: Building RpcClient with hostname: 127.0.0.1, port:
4545*
*2012-10-16 12:03:59,342 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:420)]
Batch size string = null*
*2012-10-16 12:03:59,347 (SinkRunner-PollingRunner-FailoverSinkProcessor)
[DEBUG -
org.apache.flume.sink.AvroSink.createConnection(AvroSink.java:192)] Avro
sink avroSink-1: Created RpcClient: NettyAvroRpcClient { host: localhost,
port: 4545 }*
*2012-10-16 12:04:13,849 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
Checking file:../conf/flume.conf for changes*
*2012-10-16 12:04:43,850 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
Checking file:../conf/flume.conf for changes*
*2012-10-16 12:05:13,850 (conf-file-poller-0) [DEBUG -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)]
Checking file:../conf/flume.conf for changes*

Then on the first collector

*2012-10-16 12:03:52,003 (New I/O server worker #1-2) [DEBUG -
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:244)] Avro
source avroSource-1: Received avro event batch of 100 events.*
*2012-10-16 12:03:55,004 (New I/O server worker #1-2) [ERROR -
org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:261)] Avro
source avroSource-1: Unable to process event batch. Exception follows.*
*org.apache.flume.ChannelException: Unable to put batch on required
channel: org.apache.flume.channel.MemoryChannel{name: memoryChannel-1}*
* at
org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:200)
*
* at org.apache.flume.source.AvroSource.appendBatch(AvroSource.java:259)*
* at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)*
* at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
*
* at java.lang.reflect.Method.invoke(Method.java:597)*
* at
org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
*
* at org.apache.avro.ipc.Responder.respond(Responder.java:149)*
* at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.messageReceived(NettyServer.java:188)
*
* at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
*
* at
org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.handleUpstream(NettyServer.java:173)
*
* at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
*
* at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
*
* at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
*
* at
org.jboss.netty.handler.codec.frame.FrameDecoder
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB