Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Problem with 'reload' vs 'restart' of Flume?


+
Matt Wise 2013-05-08, 19:17
+
Hari Shreedharan 2013-05-08, 19:42
+
Matt Wise 2013-05-08, 20:19
Copy link to this message
-
Re: Problem with 'reload' vs 'restart' of Flume?


--
Hari Shreedharan
On Wednesday, May 8, 2013 at 1:19 PM, Matt Wise wrote:

> Here's a dump of most of the log output for the reload process... We basically saw all log traffic stop once the reload happened. It did not resume until we did a full restart of the daemon:
>
> > 08 May 2013 19:07:05,140 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/etc/flume-ng/conf/flume.conf
> > 08 May 2013 19:07:05,141 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,141 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: s3 elasticsearch Agent: agent
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,142 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,143 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:s3
> > 08 May 2013 19:07:05,144 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:elasticsearch
> > 08 May 2013 19:07:05,152 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [agent]
At this point, it does look like your agent had started fine. I am not entirely sure why the log traffic stopped, but ti does look like the Syslog source was able to read the data. Where do you not see the data - both in HDFS and Elastic search? Can you look at hidden files in HDFS too?
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB