Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Dynamic weblog processing


Copy link to this message
-
Re: Dynamic weblog processing
navneet sharma 2012-04-09, 04:04
Thanks a lot for your responses.

I think i can start with prducer-in-app approach.

I saw the logger option as well- Kafka appender. Will it be more faster and
efficient? Anybody tried that?

On Fri, Apr 6, 2012 at 12:58 AM, Felix GV <[EMAIL PROTECTED]> wrote:

> As Hisham mentioned, what I've been working on is your option #2, and that
> can be done by using the Kafka APIs...
>
> Currently, the easiest way to get this up and running quickly would
> probably be your option #1, using the kafka-console-producer that was added
> by the kind Kafka folks in KAFKA-130 :)
>
> (It's available in 0.7)
>
> --
> Felix
>
>
>
> On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
>
> > On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> > <[EMAIL PROTECTED]> wrote:
> > > Hi,
> > >
> > > I am new to kafka. If i have an application which is generating logs
> and
> > i
> > > need to read and do some CEP on these logs.
> > >
> > > For CEP, we are planning to use Esper and for reading weblogs, we are
> > > examining Kafka. I have few questions:
> > > 1) What will be a producer in this case. Will it be an external
> > > entity(standalone program) which will do file I/o and send weblogs to
> > kafka
> > > broker?
> > > 2) Or will it be part of application itself, so that it can read logs
> in
> > > parallel with the logging utility?
> > >
> > > Any pointers will be helpful...
> > >
> > > Thanks,
> > > Navneet
> >
> > Navneet,
> >
> > We're currently doing that for our Tomcat servers. We've built an
> > access log valve (inspired by the one bundled with Tomcat that writes
> > log files to disk) that will format logs and push them into Kafka. We
> > then pull the logs and push them into HBase for analysis. FelixGV (on
> > this list) is implementing the code if you have questions for him.
> >
> > Hope this helps,
> >
> > hisham.
> >
> > --
> > Hisham Mardam Bey
> >
> > A: Because it messes up the order in which people normally read text.
> > Q: Why is top-posting such a bad thing?
> > A: Top-posting.
> > Q: What is the most annoying thing in e-mail?
> >
> > -=[ Codito Ergo Sum ]=-
> >
>