We have a use case that requires the validation of headers on events received by an avro source in order to consider an event as valid or invalid. If an event is invalid then it should be routed to a different channel.
We know how to route events based on the values of specific headers using multiplexing. However, for the regex validation of headers flume doesn't seem to provide any appropriate interceptors.
For this reason, we are thinking to create a new interceptor that would allow regex validation of headers and depending on the outcome a header would be added (e.g. valid = true)
* Does the above sound like a reasonable solution for what we want to achieve? * What would be the best way to implement it in order to be beneficial for the flume community? Extend the functionality of one of the existing interceptors (e.g. RegexFilteringInterceptor) or provide a new one?
Regards, Nikolaos http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this.
Thanks for your suggestions. We came across the morphline interceptor before but looked a little bit over complicated for what we wanted to do at that point.
However might be perfect for what we want to do now so we will give it a go.
Regards, Nikolaos ________________________________________ From: Wolfgang Hoschek [[EMAIL PROTECTED]] Sent: Thursday, February 13, 2014 8:31 PM To: [EMAIL PROTECTED] Subject: Re: Event header validation using interceptors
The morphline interceptor puts all flume events headers plus the flume event body into the input morphline record, so morphline commands can match on the entire flume event.
On Feb 13, 2014, at 9:06 PM, Jeff Lord wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext