Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Events stored in Apache Web Logs - how best to get these into Kafka?


+
James Newhaven 2013-06-11, 21:58
Copy link to this message
-
Re: Events stored in Apache Web Logs - how best to get these into Kafka?
Jun Rao 2013-06-12, 05:27
As long as you can pipe log data to console-producer, you can get the data
into Kafka.

Thanks,

Jun
On Tue, Jun 11, 2013 at 2:57 PM, James Newhaven <[EMAIL PROTECTED]>wrote:

> I have a lot of user event data being sent to an apache web server and
> written to web logs. Unfortunately I don't control this flow, but do I have
> access to the server the logs are being written to.
>
> I need a way to access these events as soon as they have been written to
> the log file and send this over to Kafka.
>
> I'm aware I can write a Kafka Producer, but I was hoping there might be
> something out of the box I could use that could monitor a log file and send
> the contents to Kafka.
>
> I saw from KAFKA-130 <https://issues.apache.org/jira/browse/KAFKA-130>
> that
> in version 0.7 onwards there is something that can monitor STDIN, not sure
> if I could use this for web log files?
>
> Thanks,
> James
>