-Re: Dynamic Key=Value Parsing with an Interceptor?
Matt Wise 2013-11-11, 18:03
Anyone have any ideas on the best way to do this?
Sr. Systems Architect
On Sat, Nov 9, 2013 at 5:28 PM, Matt Wise <[EMAIL PROTECTED]> wrote:
> Hey we'd like to set up a default format for all of our logging systems...
> perhaps looking like this:
> With this pattern, we'd allow developers to define any key/value pairs
> they want to log, and separate them with a common separator.
> If we did this, what do we need to do in Flume to get Flume to parse out
> the key=value pairs into dynamic headers? We pass our data from Flume into
> both HDFS and ElasticSearch sinks. We would really like to have these
> fields dynamically sent to the sinks for much easier parsing and analysis
> Any thoughts on this? I know that we can define a unique interceptor for
> each service that looks for explicit field names ... but thats a nightmare
> to manage. I really want something truly dynamic.
> Matt Wise
> Sr. Systems Architect