Does the Flume configuration file support parameters, for instance in the
way that you can specify them in a log4j conf file?

- Connor

Hey Connor,
Take a look at the discussion @
https://issues.apache.org/jira/browse/FLUME-1941

I you want to help work on this you are more than welcome to :)

Regards,
Mike

On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <[EMAIL PROTECTED]>wrote:

> Does the Flume configuration file support parameters, for instance in the
> way that you can specify them in a log4j conf file?
>
> - Connor
>

I'm sure this discussion has happened already, but is there a reason Spring (or something similar) isn't used for configuration, rather than the property files? It seems like there is a lot of boilerplate logic in the various implementations of Configurable.configure(Context) that is just completely unnecessary. In addition, you wouldn't have to spend time building something like property substitution that's been done many times before.
From: Mike Percy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Fri, 15 Mar 2013 10:27:00 -0700
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Parameters in Configuration File

Hey Connor,
Take a look at the discussion @ https://issues.apache.org/jira/browse/FLUME-1941

I you want to help work on this you are more than welcome to :)

Regards,
Mike

On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Does the Flume configuration file support parameters, for instance in the way that you can specify them in a log4j conf file?

- Connor

Hi Matt,
Properties files were the simple way to get started, and now that Flume is
on version 1.3.1 we can't just abandon the properties file format.

I'm open to suggestions for adopting more full featured configuration
functionality while maintaining backwards compatibility with the existing
config files.

Regards,
Mike
On Fri, Mar 15, 2013 at 10:55 AM, Kenison, Matt <[EMAIL PROTECTED]>wrote:

> I'm sure this discussion has happened already, but is there a reason
> Spring (or something similar) isn't used for configuration, rather than the
> property files? It seems like there is a lot of boilerplate logic in the
> various implementations of Configurable.configure(Context) that is just
> completely unnecessary. In addition, you wouldn't have to spend time
> building something like property substitution that's been done many times
> before.
>
>
> From: Mike Percy <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Fri, 15 Mar 2013 10:27:00 -0700
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Parameters in Configuration File
>
> Hey Connor,
> Take a look at the discussion @
> https://issues.apache.org/jira/browse/FLUME-1941
>
> I you want to help work on this you are more than welcome to :)
>
> Regards,
> Mike
>
>
>
> On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <[EMAIL PROTECTED]>wrote:
>
>> Does the Flume configuration file support parameters, for instance in the
>> way that you can specify them in a log4j conf file?
>>
>> - Connor
>>
>
>

I understand you have to continue to support existing clients. However, I don't know that total backwards compatibility would be desireable in the long run. If you want to support both the existing property configuration and a new one, one way would be to expose setters for each attribute of selectors, sinks, etc. This would let them be set directly, and allow the configuration to be moved to a separate factory class which would be used by the existing property files.

I have not dealt in detail with the configuration, though, so I'm not sure if there's something I'm overlooking.
From: Mike Percy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Fri, 15 Mar 2013 11:37:42 -0700
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Parameters in Configuration File

Hi Matt,
Properties files were the simple way to get started, and now that Flume is on version 1.3.1 we can't just abandon the properties file format.

I'm open to suggestions for adopting more full featured configuration functionality while maintaining backwards compatibility with the existing config files.

Regards,
Mike
On Fri, Mar 15, 2013 at 10:55 AM, Kenison, Matt <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
I'm sure this discussion has happened already, but is there a reason Spring (or something similar) isn't used for configuration, rather than the property files? It seems like there is a lot of boilerplate logic in the various implementations of Configurable.configure(Context) that is just completely unnecessary. In addition, you wouldn't have to spend time building something like property substitution that's been done many times before.
From: Mike Percy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Reply-To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Date: Fri, 15 Mar 2013 10:27:00 -0700
To: "[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>" <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>>
Subject: Re: Parameters in Configuration File

Hey Connor,
Take a look at the discussion @ https://issues.apache.org/jira/browse/FLUME-1941

I you want to help work on this you are more than welcome to :)

Regards,
Mike

On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Does the Flume configuration file support parameters, for instance in the way that you can specify them in a log4j conf file?

- Connor

There is some work on modifying the configuration system to make Flume work
with Zookeeper. I believe the way that's being done is by allowing for some
form of pluggable configuration system - you choose between the default
properties file or from something else. I'll look for the JIRA later; but
if you want, it should be possible to make a configuration mechanism to use
Spring instead of the properties file (I don't know enough about Spring to
say more about how it could be done).

Essentially, it should be possible to do, if you or someone else wants to
invest the time in making it work.

- Connor
On Fri, Mar 15, 2013 at 3:07 PM, Kenison, Matt <[EMAIL PROTECTED]>wrote:

> I understand you have to continue to support existing clients. However, I
> don't know that total backwards compatibility would be desireable in the
> long run. If you want to support both the existing property configuration
> and a new one, one way would be to expose setters for each attribute of
> selectors, sinks, etc. This would let them be set directly, and allow the
> configuration to be moved to a separate factory class which would be used
> by the existing property files.
>
> I have not dealt in detail with the configuration, though, so I'm not sure
> if there's something I'm overlooking.
>
>
> From: Mike Percy <[EMAIL PROTECTED]>
> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Date: Fri, 15 Mar 2013 11:37:42 -0700
>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Subject: Re: Parameters in Configuration File
>
> Hi Matt,
> Properties files were the simple way to get started, and now that Flume is
> on version 1.3.1 we can't just abandon the properties file format.
>
> I'm open to suggestions for adopting more full featured configuration
> functionality while maintaining backwards compatibility with the existing
> config files.
>
> Regards,
> Mike
>
>
> On Fri, Mar 15, 2013 at 10:55 AM, Kenison, Matt <[EMAIL PROTECTED]>wrote:
>
>> I'm sure this discussion has happened already, but is there a reason
>> Spring (or something similar) isn't used for configuration, rather than the
>> property files? It seems like there is a lot of boilerplate logic in the
>> various implementations of Configurable.configure(Context) that is just
>> completely unnecessary. In addition, you wouldn't have to spend time
>> building something like property substitution that's been done many times
>> before.
>>
>>
>> From: Mike Percy <[EMAIL PROTECTED]>
>> Reply-To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Date: Fri, 15 Mar 2013 10:27:00 -0700
>> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
>> Subject: Re: Parameters in Configuration File
>>
>> Hey Connor,
>> Take a look at the discussion @
>> https://issues.apache.org/jira/browse/FLUME-1941
>>
>> I you want to help work on this you are more than welcome to :)
>>
>> Regards,
>> Mike
>>
>>
>>
>> On Thu, Mar 14, 2013 at 5:09 PM, Connor Woodson <[EMAIL PROTECTED]>wrote:
>>
>>> Does the Flume configuration file support parameters, for instance in
>>> the way that you can specify them in a log4j conf file?
>>>
>>> - Connor
>>>
>>
>>
>