Hi All,

We have a requirement for using the tailing functionality as like Apache
Flume for log files streaming.

Can I know any built in functionality available for Kafka or any other
alternative approaches for log file streaming .

Thanks for your advanced help.

 

Thanks

Vijay
 

We don't have such built-in adaptors like Flume yet. They are very useful.
If anyone wants to work on this and contribute back, it would be great for
the community.

If you use log4j, Kafka does have a bult-in log4j appender.

Thanks,

Jun

On Wed, Feb 27, 2013 at 7:57 AM, Vijay Ram <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> We have a requirement for using the tailing functionality as like Apache
> Flume for log files streaming.
>
> Can I know any built in functionality available for Kafka or any other
> alternative approaches for log file streaming .
>
> Thanks for your advanced help.
>
>
>
> Thanks
>
> Vijay
>
>

 

Vijay,

In Kafka, we have the ConsoleConsumer that tails a topic and streams data
continuously from the logs. You can run it like this -

./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic foo

Does that help or were you referring to something else ?

Thanks,
Neha
On Wed, Feb 27, 2013 at 7:57 AM, Vijay Ram <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> We have a requirement for using the tailing functionality as like Apache
> Flume for log files streaming.
>
> Can I know any built in functionality available for Kafka or any other
> alternative approaches for log file streaming .
>
> Thanks for your advanced help.
>
>
>
> Thanks
>
> Vijay
>
>

 

Thanks Neha, Let me try as well.

Thanks
Vijay Ram

-----Original Message-----
From: Neha Narkhede [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, February 27, 2013 12:55 PM
To: [EMAIL PROTECTED]
Subject: Re: Tailing utilty in Kafka

Vijay,

In Kafka, we have the ConsoleConsumer that tails a topic and streams data
continuously from the logs. You can run it like this -

./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic foo

Does that help or were you referring to something else ?

Thanks,
Neha
On Wed, Feb 27, 2013 at 7:57 AM, Vijay Ram <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> We have a requirement for using the tailing functionality as like
> Apache Flume for log files streaming.
>
> Can I know any built in functionality available for Kafka or any other
> alternative approaches for log file streaming .
>
> Thanks for your advanced help.
>
>
>
> Thanks
>
> Vijay
>
>
 

Jun,
Sure, I will try to create similar and contribute, Let me know if any JIRA
has created.

Thanks and Regards
Vijay

-----Original Message-----
From: Jun Rao [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, February 27, 2013 12:51 PM
To: [EMAIL PROTECTED]
Subject: Re: Tailing utilty in Kafka

We don't have such built-in adaptors like Flume yet. They are very useful.
If anyone wants to work on this and contribute back, it would be great for
the community.

If you use log4j, Kafka does have a bult-in log4j appender.

Thanks,

Jun

On Wed, Feb 27, 2013 at 7:57 AM, Vijay Ram <[EMAIL PROTECTED]> wrote:

> Hi All,
>
> We have a requirement for using the tailing functionality as like
> Apache Flume for log files streaming.
>
> Can I know any built in functionality available for Kafka or any other
> alternative approaches for log file streaming .
>
> Thanks for your advanced help.
>
>
>
> Thanks
>
> Vijay
>
>
 

Vijay,

We can create a JIRA but it will help if you can describe how different it
is from ConsoleConsumer. We had intended ConsoleConsumer to be the Kafka
tail utility. It will be great
if we get feedback on how we can improve it.

Thanks,
Neha
On Wed, Feb 27, 2013 at 11:34 AM, Vijay Ram <[EMAIL PROTECTED]> wrote:

> Jun,
> Sure, I will try to create similar and contribute, Let me know if any JIRA
> has created.
>
> Thanks and Regards
> Vijay
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, February 27, 2013 12:51 PM
> To: [EMAIL PROTECTED]
> Subject: Re: Tailing utilty in Kafka
>
> We don't have such built-in adaptors like Flume yet. They are very useful.
> If anyone wants to work on this and contribute back, it would be great for
> the community.
>
> If you use log4j, Kafka does have a bult-in log4j appender.
>
> Thanks,
>
> Jun
>
> On Wed, Feb 27, 2013 at 7:57 AM, Vijay Ram <[EMAIL PROTECTED]> wrote:
>
> > Hi All,
> >
> > We have a requirement for using the tailing functionality as like
> > Apache Flume for log files streaming.
> >
> > Can I know any built in functionality available for Kafka or any other
> > alternative approaches for log file streaming .
> >
> > Thanks for your advanced help.
> >
> >
> >
> > Thanks
> >
> > Vijay
> >
> >
>
>