Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Flume >> mail # user >> Syslog Infrastructure with Flume

Josh West 2012-10-26, 14:05
Ron Thielen 2012-10-26, 21:06
Copy link to this message
Re: Syslog Infrastructure with Flume
Since you ask...

In our environment our primary concern is audit logs - have have to audit banking transactions as well as changes administrators make. We have a legacy system that needed to be integrated that had records in a form different than what we want stored. We also need to allow administrators to view events as close to real time as possible. Plus we have to aggregate data across 2 data centers. Although we are currently not including web server access logs we plan to integrate them in over time.  We also have requirements from our security team to pass events for their use to ArcSight.

1. We have a "log extractor" that receives legacy events as they occur and converts them into our new format and passes them to Flume. All new applications use the Log4j 2 Flume Appender to get data to Flume.
2. Flume passes the data to ArcSight for our security team's use.
3. We wrote a Flume to Cassandra Sink.
4. We wrote our own REST query services to retrieve the data from Cassandra.
5. Since we are using DataStax Enterprise version of Cassandra we have also set up "Analytic" nodes that run Hadoop on top of Cassandra. This allows the data to be accessed via normal Hadoop tools for data analytics.
6. We have written our own reporting UI component in our Administrative Platform to allow administrators to view activities in real time or to schedule background data collection so users can post process the data on their own.

We do not have anything to allow an admin to "tail" the log but it wouldn't be hard at all to write an application to accept Flume events via Avro and display the last "n" events as they arrive.

One thing I should point out. We format our events in accordance with RFC 5424 and store that in the Flume event body. We then store all our individual pieces of audit event data in Flume headers fields.  The RFC 5424 message is what we send to ArcSight. The event fields and the compressed body are all stored in individual columns in Cassandra.

On Oct 26, 2012, at 2:06 PM, Ron Thielen wrote:

> I am exactly where you are with this, except for the problem of my not having had time to write a serializer to address the Hostname Timestamp issue.  Questions about the use of Flume in this manner seem to recur on a regular basis, so it seems a common use case.
> Sorry I cannot offer a solution since I am in your shoes at the moment, unfortunately looking at storing logs twice.
> Ron Thielen
> <image001.jpg>
> From: Josh West [mailto:[EMAIL PROTECTED]]
> Sent: Friday, October 26, 2012 9:05 AM
> Subject: Syslog Infrastructure with Flume
> Hey folks,
> I've been experimenting with Flume for a few weeks now, trying to determine an approach to designing a reliable, highly available, scalable system to store logs from various sources, including syslog.  Ideally, this system will meet the following requirements:
> Logs from syslog across all servers make their way into HDFS.
> Logs are stored in HDFS in a manner that is available for post-processing:
> Example:  HIVE partitions - with HDFS Flume Sink, can set hdfs.path to hdfs://namenode/flume/syslog/server=%{host}/facility=%{Facility}
> Example:  Custom map reduce jobs...
> Logs are stored in HDFS in a manner that is available for "reading" by sysadmins:
> During troubleshooting/firefighting, it is quite helpful to be able to login to a central logging system and tail -f / grep logs.
> We need to be able to see the logs "live".
> Some folks may be wondering why are we choosing Flume for syslog, instead of something like Graylog2 or Logstash?  The answer is we will be using Flume + Hadoop for the transport and processing of other types of data in addition to syslog.  For example, webserver access logs for post processing and statistical analysis.  So, we would like to make the most use of the Hadoop cluster, keeping all logs of all types in one redundant/scalable solution.  Additionally, by keeping both syslog and webserver access logs in Hadoop/HDFS, we can begin to correlate events.
Roshan Naik 2012-10-29, 23:37
Josh West 2012-10-30, 09:47
Hari Shreedharan 2012-10-31, 19:22
Roshan Naik 2012-10-31, 20:31
Hari Shreedharan 2012-10-31, 20:39
Roshan Naik 2012-11-01, 18:19
Hari Shreedharan 2012-11-01, 22:52
Roshan Naik 2012-11-01, 22:54
Josh West 2012-10-30, 09:42