Hi,

I was going through archive. And came across this jira ticket:
https://issues.apache.org/jira/browse/KAFKA-130

My question, is the tailing utility available now with 0.7.0 version?
Thanks,
Navneet Sharma

Yes, in the bin/ directory you will see kafka-console-producer.sh which runs it.

-Jay

On Thu, May 10, 2012 at 11:34 PM, navneet sharma
<[EMAIL PROTECTED]> wrote:
> Hi,
>
> I was going through archive. And came across this jira ticket:
> https://issues.apache.org/jira/browse/KAFKA-130
>
> My question, is the tailing utility available now with 0.7.0 version?
>
>
> Thanks,
> Navneet Sharma

Jay,

is there any example or documentation to understand how to use it. If i
type something on console, it will be captured by producer, but how to send
messages from a syslog to this console-producer?

Thanks,
Navneet Sharma

On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Yes, in the bin/ directory you will see kafka-console-producer.sh which
> runs it.
>
> -Jay
>
> On Thu, May 10, 2012 at 11:34 PM, navneet sharma
> <[EMAIL PROTECTED]> wrote:
> > Hi,
> >
> > I was going through archive. And came across this jira ticket:
> > https://issues.apache.org/jira/browse/KAFKA-130
> >
> > My question, is the tailing utility available now with 0.7.0 version?
> >
> >
> > Thanks,
> > Navneet Sharma
>

Hi Navneet,

I don't have such an example, it would be great if somebody tried
this. I don't know whether or not this would be a good solution for
syslog, or whether it would be good to have a special-purpose daemon.
I would be happy to modify that tool if there are changes that would
be needed.

To get files instead of the console you would presumably pipe them in.

-Jay

On Tue, May 15, 2012 at 6:07 AM, navneet sharma
<[EMAIL PROTECTED]> wrote:
> Jay,
>
> is there any example or documentation to understand how to use it. If i
> type something on console, it will be captured by producer, but how to send
> messages from a syslog to this console-producer?
>
> Thanks,
> Navneet Sharma
>
> On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Yes, in the bin/ directory you will see kafka-console-producer.sh which
>> runs it.
>>
>> -Jay
>>
>> On Thu, May 10, 2012 at 11:34 PM, navneet sharma
>> <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> >
>> > I was going through archive. And came across this jira ticket:
>> > https://issues.apache.org/jira/browse/KAFKA-130
>> >
>> > My question, is the tailing utility available now with 0.7.0 version?
>> >
>> >
>> > Thanks,
>> > Navneet Sharma
>>

We built a special purpose daemon for tailing files like syslog and pumping
them into Kafka.  The reason is that reliable tailing is not trivial.  You
have to handle the rotation of syslog and other files, and be able to pick
up where you left off in case of restarts, even if the file moved -- if you
want guaranteed delivery that is.

Flume might be able to do something like this, but there are lots of
threads about how unreliable the tailing there is, and it is kind of
heavyweight, so we decided to do it ourselves.

On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Hi Navneet,
>
> I don't have such an example, it would be great if somebody tried
> this. I don't know whether or not this would be a good solution for
> syslog, or whether it would be good to have a special-purpose daemon.
> I would be happy to modify that tool if there are changes that would
> be needed.
>
> To get files instead of the console you would presumably pipe them in.
>
> -Jay
>
> On Tue, May 15, 2012 at 6:07 AM, navneet sharma
> <[EMAIL PROTECTED]> wrote:
> > Jay,
> >
> > is there any example or documentation to understand how to use it. If i
> > type something on console, it will be captured by producer, but how to
> send
> > messages from a syslog to this console-producer?
> >
> > Thanks,
> > Navneet Sharma
> >
> > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> Yes, in the bin/ directory you will see kafka-console-producer.sh which
> >> runs it.
> >>
> >> -Jay
> >>
> >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma
> >> <[EMAIL PROTECTED]> wrote:
> >> > Hi,
> >> >
> >> > I was going through archive. And came across this jira ticket:
> >> > https://issues.apache.org/jira/browse/KAFKA-130
> >> >
> >> > My question, is the tailing utility available now with 0.7.0 version?
> >> >
> >> >
> >> > Thanks,
> >> > Navneet Sharma
> >>
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Any interest in sharing what you did? Either as an example or as a
general tool...

-Jay

On Tue, May 15, 2012 at 10:07 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> We built a special purpose daemon for tailing files like syslog and pumping
> them into Kafka.  The reason is that reliable tailing is not trivial.  You
> have to handle the rotation of syslog and other files, and be able to pick
> up where you left off in case of restarts, even if the file moved -- if you
> want guaranteed delivery that is.
>
> Flume might be able to do something like this, but there are lots of
> threads about how unreliable the tailing there is, and it is kind of
> heavyweight, so we decided to do it ourselves.
>
> On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Hi Navneet,
>>
>> I don't have such an example, it would be great if somebody tried
>> this. I don't know whether or not this would be a good solution for
>> syslog, or whether it would be good to have a special-purpose daemon.
>> I would be happy to modify that tool if there are changes that would
>> be needed.
>>
>> To get files instead of the console you would presumably pipe them in.
>>
>> -Jay
>>
>> On Tue, May 15, 2012 at 6:07 AM, navneet sharma
>> <[EMAIL PROTECTED]> wrote:
>> > Jay,
>> >
>> > is there any example or documentation to understand how to use it. If i
>> > type something on console, it will be captured by producer, but how to
>> send
>> > messages from a syslog to this console-producer?
>> >
>> > Thanks,
>> > Navneet Sharma
>> >
>> > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>> >
>> >> Yes, in the bin/ directory you will see kafka-console-producer.sh which
>> >> runs it.
>> >>
>> >> -Jay
>> >>
>> >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma
>> >> <[EMAIL PROTECTED]> wrote:
>> >> > Hi,
>> >> >
>> >> > I was going through archive. And came across this jira ticket:
>> >> > https://issues.apache.org/jira/browse/KAFKA-130
>> >> >
>> >> > My question, is the tailing utility available now with 0.7.0 version?
>> >> >
>> >> >
>> >> > Thanks,
>> >> > Navneet Sharma
>> >>
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>

Yeah, one of these days we hope to open source it.  We have to first
extract a bunch of internal dependencies out, and have cycles to do it...

Basically, the idea is to monitor the file/directories where the file
rotation happens (using, say, nio2 or inotify/jnotify) and periodically
persist state for recovery.

-Evan

On Tue, May 15, 2012 at 1:18 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Any interest in sharing what you did? Either as an example or as a
> general tool...
>
> -Jay
>
> On Tue, May 15, 2012 at 10:07 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
> > We built a special purpose daemon for tailing files like syslog and
> pumping
> > them into Kafka.  The reason is that reliable tailing is not trivial.
>  You
> > have to handle the rotation of syslog and other files, and be able to
> pick
> > up where you left off in case of restarts, even if the file moved -- if
> you
> > want guaranteed delivery that is.
> >
> > Flume might be able to do something like this, but there are lots of
> > threads about how unreliable the tailing there is, and it is kind of
> > heavyweight, so we decided to do it ourselves.
> >
> > On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Navneet,
> >>
> >> I don't have such an example, it would be great if somebody tried
> >> this. I don't know whether or not this would be a good solution for
> >> syslog, or whether it would be good to have a special-purpose daemon.
> >> I would be happy to modify that tool if there are changes that would
> >> be needed.
> >>
> >> To get files instead of the console you would presumably pipe them in.
> >>
> >> -Jay
> >>
> >> On Tue, May 15, 2012 at 6:07 AM, navneet sharma
> >> <[EMAIL PROTECTED]> wrote:
> >> > Jay,
> >> >
> >> > is there any example or documentation to understand how to use it. If
> i
> >> > type something on console, it will be captured by producer, but how to
> >> send
> >> > messages from a syslog to this console-producer?
> >> >
> >> > Thanks,
> >> > Navneet Sharma
> >> >
> >> > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> >> Yes, in the bin/ directory you will see kafka-console-producer.sh
> which
> >> >> runs it.
> >> >>
> >> >> -Jay
> >> >>
> >> >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma
> >> >> <[EMAIL PROTECTED]> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I was going through archive. And came across this jira ticket:
> >> >> > https://issues.apache.org/jira/browse/KAFKA-130
> >> >> >
> >> >> > My question, is the tailing utility available now with 0.7.0
> version?
> >> >> >
> >> >> >
> >> >> > Thanks,
> >> >> > Navneet Sharma
> >> >>
> >>
> >
> >
> >
> > --
> > --
> > *Evan Chan*
> > Senior Software Engineer |
> > [EMAIL PROTECTED] | (650) 996-4600
> > www.ooyala.com | blog <http://www.ooyala.com/blog> |
> > @ooyala<http://www.twitter.com/ooyala>
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Yeah, that would be a great tool to have out there if you guys ever
get around to it...

-Jay

On Tue, May 15, 2012 at 1:23 PM, Evan Chan <[EMAIL PROTECTED]> wrote:
> Yeah, one of these days we hope to open source it.  We have to first
> extract a bunch of internal dependencies out, and have cycles to do it...
>
> Basically, the idea is to monitor the file/directories where the file
> rotation happens (using, say, nio2 or inotify/jnotify) and periodically
> persist state for recovery.
>
> -Evan
>
> On Tue, May 15, 2012 at 1:18 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>
>> Any interest in sharing what you did? Either as an example or as a
>> general tool...
>>
>> -Jay
>>
>> On Tue, May 15, 2012 at 10:07 AM, Evan Chan <[EMAIL PROTECTED]> wrote:
>> > We built a special purpose daemon for tailing files like syslog and
>> pumping
>> > them into Kafka.  The reason is that reliable tailing is not trivial.
>>  You
>> > have to handle the rotation of syslog and other files, and be able to
>> pick
>> > up where you left off in case of restarts, even if the file moved -- if
>> you
>> > want guaranteed delivery that is.
>> >
>> > Flume might be able to do something like this, but there are lots of
>> > threads about how unreliable the tailing there is, and it is kind of
>> > heavyweight, so we decided to do it ourselves.
>> >
>> > On Tue, May 15, 2012 at 8:22 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>> >
>> >> Hi Navneet,
>> >>
>> >> I don't have such an example, it would be great if somebody tried
>> >> this. I don't know whether or not this would be a good solution for
>> >> syslog, or whether it would be good to have a special-purpose daemon.
>> >> I would be happy to modify that tool if there are changes that would
>> >> be needed.
>> >>
>> >> To get files instead of the console you would presumably pipe them in.
>> >>
>> >> -Jay
>> >>
>> >> On Tue, May 15, 2012 at 6:07 AM, navneet sharma
>> >> <[EMAIL PROTECTED]> wrote:
>> >> > Jay,
>> >> >
>> >> > is there any example or documentation to understand how to use it. If
>> i
>> >> > type something on console, it will be captured by producer, but how to
>> >> send
>> >> > messages from a syslog to this console-producer?
>> >> >
>> >> > Thanks,
>> >> > Navneet Sharma
>> >> >
>> >> > On Sun, May 13, 2012 at 2:26 AM, Jay Kreps <[EMAIL PROTECTED]>
>> wrote:
>> >> >
>> >> >> Yes, in the bin/ directory you will see kafka-console-producer.sh
>> which
>> >> >> runs it.
>> >> >>
>> >> >> -Jay
>> >> >>
>> >> >> On Thu, May 10, 2012 at 11:34 PM, navneet sharma
>> >> >> <[EMAIL PROTECTED]> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > I was going through archive. And came across this jira ticket:
>> >> >> > https://issues.apache.org/jira/browse/KAFKA-130
>> >> >> >
>> >> >> > My question, is the tailing utility available now with 0.7.0
>> version?
>> >> >> >
>> >> >> >
>> >> >> > Thanks,
>> >> >> > Navneet Sharma
>> >> >>
>> >>
>> >
>> >
>> >
>> > --
>> > --
>> > *Evan Chan*
>> > Senior Software Engineer |
>> > [EMAIL PROTECTED] | (650) 996-4600
>> > www.ooyala.com | blog <http://www.ooyala.com/blog> |
>> > @ooyala<http://www.twitter.com/ooyala>
>>
>
>
>
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> [EMAIL PROTECTED] | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>