Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Problem with 'reload' vs 'restart' of Flume?


Copy link to this message
-
Re: Problem with 'reload' vs 'restart' of Flume?


--
Hari Shreedharan
On Wednesday, May 8, 2013 at 1:19 PM, Matt Wise wrote:

> Here's a dump of most of the log output for the reload process... We basically saw all log traffic stop once the reload happened. It did not resume until we did a full restart of the daemon:
>
> > 08 May 2013 19:07:05,140 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 08 May 2013 19:07:05,141 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,141 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: s3 elasticsearch Agent: agent
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,152 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [agent]
At this point, it does look like your agent had started fine. I am not entirely sure why the log traffic stopped, but ti does look like the Syslog source was able to read the data. Where do you not see the data - both in HDFS and Elastic search? Can you look at hidden files in HDFS too?