Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # dev >> Review Request: FLUME-1772: AbstractConfigurationProvider should remove component which throws exception from configure method.


Copy link to this message
-
Re: Review Request: FLUME-1772: AbstractConfigurationProvider should remove component which throws exception from configure method.


> On Dec. 8, 2012, 12:39 a.m., Hari Shreedharan wrote:
> > Generally looks good. I have a couple of comments:
> > * There are a lot of tabs/trailing spaces. Could you please remove those.
> > * If we remove these components, we need to make sure that after all components are loaded/removed, we have at least 1 channel with a source or sink in the agent. Basically after all load* methods are done, we should iterate through the sinks and sources and verify their channels(in case of sources, at least one of the channels) are still around (if not the source/sink should be removed), else the config is invalid.
>
> Hari Shreedharan wrote:
>     Also after iterating, there must be at least one source or sink remaining.
>
> Brock Noland wrote:
>     Thanks Hari, sorry for all the spaces. I thought I had removed them.
>    
>     Regarding the sources/sinks. That is a good point....I wonder about implementation though. For example, often for testing we might configure a source and channel with no sink. Or in production after having some issues with a file channel the data may be moved aside and after some debugging a sink and channel hooked together to put the data back into the system.  What are your thoughts about these cases?

I would assume it should be something like this:

* Iterate through the channels and configure them - removing any badly configured ones
* Iterate through sources - first check if the channel is still around, if it is not - remove, else configure. If it throws exception, remove it. Same for sinks.
* Now iterate through the channels once again to check there is at least one sink or source (maybe we should keep a counter or map of components connected to the channels)
* Finally, verify there is at least one channel (by induction, there is at least one sink or source connected to it).

FYI - Something similar is done in FlumeConfiguration.java, but not exactly the same though.
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8421/#review14202
-----------------------------------------------------------
On Dec. 7, 2012, 8:37 p.m., Brock Noland wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/8421/
> -----------------------------------------------------------
>
> (Updated Dec. 7, 2012, 8:37 p.m.)
>
>
> Review request for Flume.
>
>
> Description
> -------
>
> Catches Exception around all the configure() methods of components. If they throw an exception, they are removed from the configuration (not added).
>
>
> This addresses bug FLUME-1772.
>     https://issues.apache.org/jira/browse/FLUME-1772
>
>
> Diffs
> -----
>
>   flume-ng-node/src/main/java/org/apache/flume/node/AbstractConfigurationProvider.java daef76b
>   flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractConfigurationProvider.java 25001b1
>
> Diff: https://reviews.apache.org/r/8421/diff/
>
>
> Testing
> -------
>
> Tests added for source, channel, and sink.
>
>
> Thanks,
>
> Brock Noland
>
>