Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Re: Automatic log analysis and alert generation


Copy link to this message
-
Re: Automatic log analysis and alert generation
Mark,

FWIW, here I'm using flume to pull all the logs to a central server and
then using a package called logcheck to scan the logs for keywords/events
periodically.  A bit of perl scripting to select the logs I want and such,
I just have it run every 30 minutes as that is frequent enough for my
purposes.  I'm scanning about 100 log files (most pretty active) and
runtime is about 2 mins with my crude perl scripting.

There are likely other packages out there that do things in a similar
fashion, but the logcheck scripts have served me well for over 15 years, so
to say the least it's inner workings are familiar. Might be worth looking
at.

Dave

On Mon, Aug 26, 2013 at 3:22 PM, Mark Nuttall-Smith <
[EMAIL PROTECTED]> wrote:

> Hi, I posted this question on stackoverflow (
> http://stackoverflow.com/questions/18448218/automatic-log-analysis-and-alert-generation),
> but thought I might get a better response here, so am crossposting... hope
> it's ok!
>
>  I would like some design advice for a centralized logging project I am
> considering. I have a number of components producing logs on various
> servers. Apache Flume looks like the sensible choice for streaming to a
> central log server, most likely into an elasticsearch instance for querying
> and analysis.
>
> Here's my question: I would like to provide a scripting engine listening
> to the flow of log events arriving on the central server. Would it make
> sense to do that as an interceptor in Flume, or as a plugin to
> elasticsearch, or something else completely?
>
> Thanks,
>
> Mark
>
>
+
Wolfgang Hoschek 2013-08-26, 21:12
+
Mark Nuttall-Smith 2013-08-28, 07:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB