Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Problem with 'reload' vs 'restart' of Flume?


Copy link to this message
-
Problem with 'reload' vs 'restart' of Flume?
We're seeing problems when we try to live-reload our Flume agents rather than restart them. They seem to maintain their incoming Syslog connections from the clients, but they stop sending out data to ElasticSearch (and probably the HDFS plugin as well). I see these errors during the reload, and I'm wondering if they're related. The process to reproduce for us is to make any change to the flume.conf file, and wait until Flume detects the file change. When this happens, everything basically breaks.

08 May 2013 19:07:32,413 ERROR [lifecycleSupervisor-1-6] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:92)  - Failed to register monitored counter group for type: CHANNEL, name: fc1
javax.management.InstanceAlreadyExistsException: org.apache.flume.channel:type=fc1
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
at org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:87)
at org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:67)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:323)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)

08 May 2013 19:07:32,418 INFO  [lifecycleSupervisor-1-8] (org.apache.flume.sink.elasticsearch.ElasticSearchSink.start:319)  - ElasticSearch sink {} started
08 May 2013 19:07:32,418 ERROR [lifecycleSupervisor-1-8] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:92)  - Failed to register monitored counter group for type: SINK, name: elasticsearch
javax.management.InstanceAlreadyExistsException: org.apache.flume.sink:type=elasticsearch
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
at org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:87)
at org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:67)
at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:320)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
08 May 2013 19:07:32,426 INFO  [lifecycleSupervisor-1-8] (org.apache.flume.instrumentation.MonitoredCounterGroup.start:73)  - Component type: SINK, name: elasticsearch started