Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Flume >> mail # user >> Re: Automatic log analysis and alert generation


Copy link to this message
-
Re: Automatic log analysis and alert generation
Take a look at the Apache Flume Morphline Solr Sink, for example for heavy duty ETL processing and
ingestion into Solr:

       http://flume.apache.org/FlumeUserGuide.html#morphlinesolrsink

It provides a scripting engine that enables CEP on the flow of log events.

Wolfgang.

On Aug 26, 2013, at 1:22 PM, Mark Nuttall-Smith wrote:

> Hi, I posted this question on stackoverflow (http://stackoverflow.com/questions/18448218/automatic-log-analysis-and-alert-generation), but thought I might get a better response here, so am crossposting... hope it's ok!
>
> I would like some design advice for a centralized logging project I am considering. I have a number of components producing logs on various servers. Apache Flume looks like the sensible choice for streaming to a central log server, most likely into an elasticsearch instance for querying and analysis.
>
> Here's my question: I would like to provide a scripting engine listening to the flow of log events arriving on the central server. Would it make sense to do that as an interceptor in Flume, or as a plugin to elasticsearch, or something else completely?
>
> Thanks,
>
> Mark
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB