Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka syslog producer


Copy link to this message
-
Re: Kafka syslog producer
I am working on a similar use-case.
I think there are 2 approaches which you can think of:
1) Application generating syslog should have a parallel producer to send
syslog to kafka broker.
2) try the kafka log4j appender, but again it will be in parallel with
syslogs

Alternatively, i am trying with file APIs - infinite while loop on the
syslog file, but that is very inefficient.

-navneet sharma

On Tue, Apr 24, 2012 at 1:31 AM, Flinkster <[EMAIL PROTECTED]> wrote:

> How does LinkedIn handle syslog? Is Kafka at LinkedIN dedicated to
> application logs & metrics and syslog messages (e.g. network, OS logs)
> is a separate flow and datastore?
>
> On Mon, Apr 23, 2012 at 11:26 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
> > This is an interesting use-case and similar to using the log4j appender
> for
> > application logs. What you describe sounds reasonable: i.e., have a
> > producer process on the syslog server to send syslog messages to your
> Kafka
> > brokers.
> >
> >
> >> 2) How about tailing a file to a central logging like you can do with
> >> scribe/flume agents?
> >>
> >
> > You could use tail and pipe (or a named pipe) to a console-based
> producer.
> > I don't know enough about syslog.d to tell if this would be too much of a
> > hack or not, but I think it would be more reliable than using tail+pipes:
> > you could configure your local machines to direct syslog messages to the
> > remote server and you could write a simple component that listens on the
> > syslog socket and redirects incoming messages to a Kafka producer.
> >
> > Joel
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB