Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume, mail # user - Problem with 'reload' vs 'restart' of Flume?


Copy link to this message
-
Re: Problem with 'reload' vs 'restart' of Flume?
Matt Wise 2013-05-08, 20:19
Here's a dump of most of the log output for the reload process... We basically saw all log traffic stop once the reload happened. It did not resume until we did a full restart of the daemon:

> 08 May 2013 19:07:05,140 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> 08 May 2013 19:07:05,141 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,141 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: s3 elasticsearch Agent: agent
> 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> 08 May 2013 19:07:05,152 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [agent]

.. at this point we restarted the agent entirely...

here's our flume agent config:
On May 8, 2013, at 12:42 PM, Hari Shreedharan <[EMAIL PROTECTED]> wrote: