Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume, mail # user - Re: Flume-ng InstanceAlreadyExistsException


+
Venkatesh S R 2013-09-05, 18:15
Copy link to this message
-
Re: Flume-ng InstanceAlreadyExistsException
Deepesh Malviya 2013-09-06, 04:39
Venkatesh, Thanks for the reply.

My other question was the latter one as you mentioned in your reply.
Our each sinks are tied to the file channel in our configuration i.e. each
sink will have its own file channel. The number sinks will be more the
number of threads (/cores) available & I think the maximum number of
channel/sink we will hit will be 99. I hope flume-ng handles it nicely.

How does flume-ng handles incoming events when it has reached the full
memory allocated to it? Does it exit or does it rejects & return error code
to the downstream agents?

Thanks,
Deepesh
On Thu, Sep 5, 2013 at 11:45 PM, Venkatesh S R <[EMAIL PROTECTED]> wrote:

> Hi Deepesh: I think you can ignore the exception stack, its benign. Flume
> tries to re-register the monitoring MBean whenever the configurations are
> dynamically refreshed and it re-initializes the components. And because the
> MBean was already registered, you see this exception trace. I remember
> there was a Jira ticket to suppress this and Mike/Hari fixed the same in
> trunk.
>
> On the other question I assume you are either asking if there a minimum of
> the number sinks required or if there is limit to the total number of sinks
> in an instance, for both questions the answer is NO.
>
> I would think of the heap size proportional to the Channel's (Memory) size
> you are setting up, to make sure there is enough space to hold all the
> events when required. The number of sinks that are setup, would relate to
> the number of threads (/cores) available for you to use.
>
> Other folks can chime in and add as required.
>
> Thanks
> Venkatesh
>
>
>
> On Wed, Sep 4, 2013 at 11:54 PM, Deepesh Malviya <[EMAIL PROTECTED]>wrote:
>
>> Hi,
>>
>> We have configured flume-ng in our environment & we dynamically update
>> the configuration. In our case, we will be having 20+ sinks & will increase
>> as we update the conf. The problem we are seeing in it is the following
>> exception is coming repeatedly for most of the configured sinks
>>
>>
>> 05 Sep 2013 05:00:38,770 ERROR [lifecycleSupervisor-1-3]
>> (org.apache.flume.instrumentation.MonitoredCounterGroup.register:113)  -
>> Failed to register monitored counter group for type: SINK, name: serviceSink
>> javax.management.InstanceAlreadyExistsException:
>> org.apache.flume.sink:type=serviceSink
>> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:467)
>>  at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1520)
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:986)
>>  at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:938)
>> at
>> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:330)
>>  at
>> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:517)
>> at
>> org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:108)
>>  at
>> org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:88)
>> at org.apache.flume.sink.hdfs.HDFSEventSink.start(HDFSEventSink.java:484)
>>  at
>> org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
>> at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
>>  at
>> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>  at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>>  at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
>>  at
>

_Deepesh