Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Events stored in Apache Web Logs - how best to get these into Kafka?


+
James Newhaven 2013-06-11, 21:58
Copy link to this message
-
Re: Events stored in Apache Web Logs - how best to get these into Kafka?
As long as you can pipe log data to console-producer, you can get the data
into Kafka.

Thanks,

Jun
On Tue, Jun 11, 2013 at 2:57 PM, James Newhaven <[EMAIL PROTECTED]>wrote:

> I have a lot of user event data being sent to an apache web server and
> written to web logs. Unfortunately I don't control this flow, but do I have
> access to the server the logs are being written to.
>
> I need a way to access these events as soon as they have been written to
> the log file and send this over to Kafka.
>
> I'm aware I can write a Kafka Producer, but I was hoping there might be
> something out of the box I could use that could monitor a log file and send
> the contents to Kafka.
>
> I saw from KAFKA-130 <https://issues.apache.org/jira/browse/KAFKA-130>
> that
> in version 0.7 onwards there is something that can monitor STDIN, not sure
> if I could use this for web log files?
>
> Thanks,
> James
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB