-Re: New Features Proposed for Apache Flume
Wolfgang Hoschek 2013-08-28, 17:21
This functionality is already available in the form of the Apache Flume MorphlineInterceptor  with the grok command . While grok is very useful, consider that grok alone often isn't enough - you typically need some other log event processing commands as well, for example as contained in morphlines .
True file tailing would be great.
Merging multiple lines into one event can already be done with the MorphlineInterceptor with the readMultiLine command . Or maybe embed a morphline directly into that new FileSource?
Seems to me that it would be more flexible, powerful and reusable to add this kind of functionality as a morphline command - contributions welcome!
Finally, a word of caution, Maxmind is a good geo db, and I've used it before, but it has some LGPL issues that may or may not be workable in this context. Maxmind db fits into RAM - Lucene seems like overkill here - you can do fast maxmind lookups directly without Lucene.
> Using the Tailer feature from Apache Commons I/O utility , we can tail
> specific files for events.
> This allows us to, regardless of the operating system, have the ability to
> watch files for future events as they occur.
> It also allows us to step in and determine if two or more events should be
> merged into one events if newline characters are present in an event.
> We can configure certain regular expressions that determines if a specific
> line is a new event or part of the prevent event.
> Essentially, this source will have the ability to merge multiple lines into
> one event before it is passed on to interceptors.
> It has been complicated group multiple lines into a single event with the
> Spooling Directory Source or Exec Source. I tried creating custom
> deserializers but it was hard to get around the logic used to parse the
> Using the Spooling Directory also means we cannot watch the original files
> so we need a background process to copy over the log files into the
> spooling directory which requires additional setup.
> The tail command is not also available on all operating systems out of the
> With this interceptor we can parse semi-structure and unstructured text and
> log data in the headers and body of the event into something structured
> that can be easily queried.
> I plan to use the information  and  for this.
> With this interceptor, we can extract HTTP response codes, response times,
> user agents, IP addresses and a whole bunch of useful data point from free
> form text.
> This is for IP intelligence.
> This interceptor will allow us to use the value of an IP address in the
> event header or body of the request to estimate the geographical location
> of the IP address.
> Using the database available here , we can inject the two-letter code or
> country name of the IP address into the event.
> We can also deduce other values such as city name, postalCode, latitude,
> longitude, Internet Service Provider and Organization name.
> This can be very helpful in analyzing traffic patterns and target audience
> from webserver or application logs.
> The database is loaded into a Lucene index when the agent is started up.
> The index is only created once if it does not already exists.
> As the interceptor comes across events, it maps the IP address to a variety
> of values that can be injected into the events.
> This can provide another option for setting up a fan-in and/or fan-out