-Re: Dynamic weblog processing
navneet sharma 2012-04-09, 04:04
Thanks a lot for your responses.
I think i can start with prducer-in-app approach.
I saw the logger option as well- Kafka appender. Will it be more faster and
efficient? Anybody tried that?
On Fri, Apr 6, 2012 at 12:58 AM, Felix GV <[EMAIL PROTECTED]> wrote:
> As Hisham mentioned, what I've been working on is your option #2, and that
> can be done by using the Kafka APIs...
> Currently, the easiest way to get this up and running quickly would
> probably be your option #1, using the kafka-console-producer that was added
> by the kind Kafka folks in KAFKA-130 :)
> (It's available in 0.7)
> On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> > On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> > <[EMAIL PROTECTED]> wrote:
> > > Hi,
> > >
> > > I am new to kafka. If i have an application which is generating logs
> > i
> > > need to read and do some CEP on these logs.
> > >
> > > For CEP, we are planning to use Esper and for reading weblogs, we are
> > > examining Kafka. I have few questions:
> > > 1) What will be a producer in this case. Will it be an external
> > > entity(standalone program) which will do file I/o and send weblogs to
> > kafka
> > > broker?
> > > 2) Or will it be part of application itself, so that it can read logs
> > > parallel with the logging utility?
> > >
> > > Any pointers will be helpful...
> > >
> > > Thanks,
> > > Navneet
> > Navneet,
> > We're currently doing that for our Tomcat servers. We've built an
> > access log valve (inspired by the one bundled with Tomcat that writes
> > log files to disk) that will format logs and push them into Kafka. We
> > then pull the logs and push them into HBase for analysis. FelixGV (on
> > this list) is implementing the code if you have questions for him.
> > Hope this helps,
> > hisham.
> > --
> > Hisham Mardam Bey
> > A: Because it messes up the order in which people normally read text.
> > Q: Why is top-posting such a bad thing?
> > A: Top-posting.
> > Q: What is the most annoying thing in e-mail?
> > -=[ Codito Ergo Sum ]=-