Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Dynamic weblog processing


Copy link to this message
-
Re: Dynamic weblog processing
As Hisham mentioned, what I've been working on is your option #2, and that
can be done by using the Kafka APIs...

Currently, the easiest way to get this up and running quickly would
probably be your option #1, using the kafka-console-producer that was added
by the kind Kafka folks in KAFKA-130 :)

(It's available in 0.7)

--
Felix

On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:

> On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I am new to kafka. If i have an application which is generating logs and
> i
> > need to read and do some CEP on these logs.
> >
> > For CEP, we are planning to use Esper and for reading weblogs, we are
> > examining Kafka. I have few questions:
> > 1) What will be a producer in this case. Will it be an external
> > entity(standalone program) which will do file I/o and send weblogs to
> kafka
> > broker?
> > 2) Or will it be part of application itself, so that it can read logs in
> > parallel with the logging utility?
> >
> > Any pointers will be helpful...
> >
> > Thanks,
> > Navneet
>
> Navneet,
>
> We're currently doing that for our Tomcat servers. We've built an
> access log valve (inspired by the one bundled with Tomcat that writes
> log files to disk) that will format logs and push them into Kafka. We
> then pull the logs and push them into HBase for analysis. FelixGV (on
> this list) is implementing the code if you have questions for him.
>
> Hope this helps,
>
> hisham.
>
> --
> Hisham Mardam Bey
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB