Hi,

I am new to kafka. If i have an application which is generating logs and i
need to read and do some CEP on these logs.

For CEP, we are planning to use Esper and for reading weblogs, we are
examining Kafka. I have few questions:
1) What will be a producer in this case. Will it be an external
entity(standalone program) which will do file I/o and send weblogs to kafka
broker?
2) Or will it be part of application itself, so that it can read logs in
parallel with the logging utility?

Any pointers will be helpful...

Thanks,
Navneet

Navneet,

Either way is fine. At LinkedIn, each app directly writes log data to a
kafka cluster using the producer api.

Thanks,

Jun

On Wed, Apr 4, 2012 at 2:20 AM, navneet sharma
<[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am new to kafka. If i have an application which is generating logs and i
> need to read and do some CEP on these logs.
>
> For CEP, we are planning to use Esper and for reading weblogs, we are
> examining Kafka. I have few questions:
> 1) What will be a producer in this case. Will it be an external
> entity(standalone program) which will do file I/o and send weblogs to kafka
> broker?
> 2) Or will it be part of application itself, so that it can read logs in
> parallel with the logging utility?
>
> Any pointers will be helpful...
>
> Thanks,
> Navneet
>

On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I am new to kafka. If i have an application which is generating logs and i
> need to read and do some CEP on these logs.
>
> For CEP, we are planning to use Esper and for reading weblogs, we are
> examining Kafka. I have few questions:
> 1) What will be a producer in this case. Will it be an external
> entity(standalone program) which will do file I/o and send weblogs to kafka
> broker?
> 2) Or will it be part of application itself, so that it can read logs in
> parallel with the logging utility?
>
> Any pointers will be helpful...
>
> Thanks,
> Navneet

Navneet,

We're currently doing that for our Tomcat servers. We've built an
access log valve (inspired by the one bundled with Tomcat that writes
log files to disk) that will format logs and push them into Kafka. We
then pull the logs and push them into HBase for analysis. FelixGV (on
this list) is implementing the code if you have questions for him.

Hope this helps,

hisham.

--
Hisham Mardam Bey

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

-=[ Codito Ergo Sum ]=-

As Hisham mentioned, what I've been working on is your option #2, and that
can be done by using the Kafka APIs...

Currently, the easiest way to get this up and running quickly would
probably be your option #1, using the kafka-console-producer that was added
by the kind Kafka folks in KAFKA-130 :)

(It's available in 0.7)

--
Felix

On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]>wrote:

> On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I am new to kafka. If i have an application which is generating logs and
> i
> > need to read and do some CEP on these logs.
> >
> > For CEP, we are planning to use Esper and for reading weblogs, we are
> > examining Kafka. I have few questions:
> > 1) What will be a producer in this case. Will it be an external
> > entity(standalone program) which will do file I/o and send weblogs to
> kafka
> > broker?
> > 2) Or will it be part of application itself, so that it can read logs in
> > parallel with the logging utility?
> >
> > Any pointers will be helpful...
> >
> > Thanks,
> > Navneet
>
> Navneet,
>
> We're currently doing that for our Tomcat servers. We've built an
> access log valve (inspired by the one bundled with Tomcat that writes
> log files to disk) that will format logs and push them into Kafka. We
> then pull the logs and push them into HBase for analysis. FelixGV (on
> this list) is implementing the code if you have questions for him.
>
> Hope this helps,
>
> hisham.
>
> --
> Hisham Mardam Bey
>
> A: Because it messes up the order in which people normally read text.
> Q: Why is top-posting such a bad thing?
> A: Top-posting.
> Q: What is the most annoying thing in e-mail?
>
> -=[ Codito Ergo Sum ]=-
>

Thanks a lot for your responses.

I think i can start with prducer-in-app approach.

I saw the logger option as well- Kafka appender. Will it be more faster and
efficient? Anybody tried that?

On Fri, Apr 6, 2012 at 12:58 AM, Felix GV <[EMAIL PROTECTED]> wrote:

> As Hisham mentioned, what I've been working on is your option #2, and that
> can be done by using the Kafka APIs...
>
> Currently, the easiest way to get this up and running quickly would
> probably be your option #1, using the kafka-console-producer that was added
> by the kind Kafka folks in KAFKA-130 :)
>
> (It's available in 0.7)
>
> --
> Felix
>
>
>
> On Wed, Apr 4, 2012 at 9:56 PM, Hisham Mardam-Bey <[EMAIL PROTECTED]
> >wrote:
>
> > On Wed, Apr 4, 2012 at 5:20 AM, navneet sharma
> > <[EMAIL PROTECTED]> wrote:
> > > Hi,
> > >
> > > I am new to kafka. If i have an application which is generating logs
> and
> > i
> > > need to read and do some CEP on these logs.
> > >
> > > For CEP, we are planning to use Esper and for reading weblogs, we are
> > > examining Kafka. I have few questions:
> > > 1) What will be a producer in this case. Will it be an external
> > > entity(standalone program) which will do file I/o and send weblogs to
> > kafka
> > > broker?
> > > 2) Or will it be part of application itself, so that it can read logs
> in
> > > parallel with the logging utility?
> > >
> > > Any pointers will be helpful...
> > >
> > > Thanks,
> > > Navneet
> >
> > Navneet,
> >
> > We're currently doing that for our Tomcat servers. We've built an
> > access log valve (inspired by the one bundled with Tomcat that writes
> > log files to disk) that will format logs and push them into Kafka. We
> > then pull the logs and push them into HBase for analysis. FelixGV (on
> > this list) is implementing the code if you have questions for him.
> >
> > Hope this helps,
> >
> > hisham.
> >
> > --
> > Hisham Mardam Bey
> >
> > A: Because it messes up the order in which people normally read text.
> > Q: Why is top-posting such a bad thing?
> > A: Top-posting.
> > Q: What is the most annoying thing in e-mail?
> >
> > -=[ Codito Ergo Sum ]=-
> >
>