Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Parameters in Configuration File


+
Connor Woodson 2013-03-15, 00:09
+
Mike Percy 2013-03-15, 17:27
+
Kenison, Matt 2013-03-15, 17:55
+
Mike Percy 2013-03-15, 18:37
+
Kenison, Matt 2013-03-15, 22:07
Copy link to this message
-
Re: Parameters in Configuration File
There is some work on modifying the configuration system to make Flume work
with Zookeeper. I believe the way that's being done is by allowing for some
form of pluggable configuration system - you choose between the default
properties file or from something else. I'll look for the JIRA later; but
if you want, it should be possible to make a configuration mechanism to use
Spring instead of the properties file (I don't know enough about Spring to
say more about how it could be done).

Essentially, it should be possible to do, if you or someone else wants to
invest the time in making it work.

- Connor
On Fri, Mar 15, 2013 at 3:07 PM, Kenison, Matt <[EMAIL PROTECTED]>wrote:

> I understand you have to continue to support existing clients. However, I
> don't know that total backwards compatibility would be desireable in the
> long run. If you want to support both the existing property configuration
> and a new one, one way would be to expose setters for each attribute of
> selectors, sinks, etc. This would let them be set directly, and allow the
> configuration to be moved to a separate factory class which would be used
> by the existing property files.
>
> I have not dealt in detail with the configuration, though, so I'm not sure
> if there's something I'm overlooking.
>
>
> From: Mike Percy <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Fri, 15 Mar 2013 11:37:42 -0700
>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Parameters in Configuration File
>
> Hi Matt,
> Properties files were the simple way to get started, and now that Flume is
> on version 1.3.1 we can't just abandon the properties file format.
>
> I'm open to suggestions for adopting more full featured configuration
> functionality while maintaining backwards compatibility with the existing
> config files.
>
> Regards,
> Mike
>
>
> On Fri, Mar 15, 2013 at 10:55 AM, Kenison, Matt <[EMAIL PROTECTED]>wrote:
>
>> I'm sure this discussion has happened already, but is there a reason
>> Spring (or something similar) isn't used for configuration, rather than the
>> property files? It seems like there is a lot of boilerplate logic in the
>> various implementations of Configurable.configure(Context) that is just
>> completely unnecessary. In addition, you wouldn't have to spend time
>> building something like property substitution that's been done many times
>> before.
>>
>>
>> From: Mike Percy <[EMAIL PROTECTED]>
>> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Date: Fri, 15 Mar 2013 10:27:00 -0700
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Re: Parameters in Configuration File
>>
>> Hey Connor,
>> Take a look at the discussion @
>> https://issues.apache.org/jira/browse/FLUME-1941
>>
>> I you want to help work on this you are more than welcome to :)
>>
>> Regards,
>> Mike
>>
>>
>>
>> On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <[EMAIL PROTECTED]>wrote:
>>
>>> Does the Flume configuration file support parameters, for instance in
>>> the way that you can specify them in a log4j conf file?
>>>
>>> - Connor
>>>
>>
>>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB