Are there any similar thing like the Flume agent such that they are
installed at the place where the logs was produced, and temporarily
store the message so as to reduce the network latency during every API
calls to the Kafka's broker?

Thanks.

Howard,

Kafka consumers fetch data in bulk, controlled by the consumer's
fetch.size, to reduce the overhead of a network roundtrip. Hence,
there is no need for any special agent between a Kafka broker and
consumer.

Thanks,
Neha

On Tue, Oct 9, 2012 at 8:48 AM, howard chen <[EMAIL PROTECTED]> wrote:
> Are there any similar thing like the Flume agent such that they are
> installed at the place where the logs was produced, and temporarily
> store the message so as to reduce the network latency during every API
> calls to the Kafka's broker?
>
> Thanks.

Hi,
On Tue, Oct 9, 2012 at 11:51 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
> Howard,
>
> Kafka consumers fetch data in bulk, controlled by the consumer's
> fetch.size, to reduce the overhead of a network roundtrip. Hence,
> there is no need for any special agent between a Kafka broker and
> consumer.

My concern is on the producer hosts, i.e. calling localhost should
minimize the network latency, similar to a localhost SMTP relay.

Thanks.

Yeah I think this is a reasonable strategy for reducing client side logic.
For the java client this is not needed as we have an async mode meant
specifically for this that will batch messages together and send out
batched requests in a background thread.

I am not aware of anyone having open sourced what you describe. If anyone
is interested one nice way to implement this would be to implement a daemon
that could interact with the syslogd protocol over a unix domain socket.
Many scripting languages have good syslogd integration and this would
probably be a good strategy for supporting logging from this kind of
process.

-Jay

On Tue, Oct 9, 2012 at 8:53 AM, howard chen <[EMAIL PROTECTED]> wrote:

> Hi,
>
>
> On Tue, Oct 9, 2012 at 11:51 PM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> > Howard,
> >
> > Kafka consumers fetch data in bulk, controlled by the consumer's
> > fetch.size, to reduce the overhead of a network roundtrip. Hence,
> > there is no need for any special agent between a Kafka broker and
> > consumer.
>
> My concern is on the producer hosts, i.e. calling localhost should
> minimize the network latency, similar to a localhost SMTP relay.
>
> Thanks.
>

Hi,

On Wed, Oct 10, 2012 at 12:20 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> I am not aware of anyone having open sourced what you describe. If anyone
> is interested one nice way to implement this would be to implement a daemon
> that could interact with the syslogd protocol over a unix domain socket.
> Many scripting languages have good syslogd integration and this would
> probably be a good strategy for supporting logging from this kind of
> process.
>

In the simplest way, able to tail a file would be enough for this purpose.

Existing app keep writing to local log file, no any blocking and not
affected by network connection, and another process tail the file and
send the changes to broker.

Scribe and Flume also have this build in, would be nice if Kafka also
support this out of the box.

(From old posts it was said that Window does not have tail...well, I
think most people run on Linux anyway..)

Thanks.

Yes, perhaps the existing console consumer could be modified to do this?

-Jay

On Tue, Oct 9, 2012 at 10:05 PM, howard chen <[EMAIL PROTECTED]> wrote:

> Hi,
>
> On Wed, Oct 10, 2012 at 12:20 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > I am not aware of anyone having open sourced what you describe. If anyone
> > is interested one nice way to implement this would be to implement a
> daemon
> > that could interact with the syslogd protocol over a unix domain socket.
> > Many scripting languages have good syslogd integration and this would
> > probably be a good strategy for supporting logging from this kind of
> > process.
> >
>
> In the simplest way, able to tail a file would be enough for this purpose.
>
> Existing app keep writing to local log file, no any blocking and not
> affected by network connection, and another process tail the file and
> send the changes to broker.
>
> Scribe and Flume also have this build in, would be nice if Kafka also
> support this out of the box.
>
> (From old posts it was said that Window does not have tail...well, I
> think most people run on Linux anyway..)
>
> Thanks.
>

You mean the producer?

On Wed, Oct 10, 2012 at 5:23 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> Yes, perhaps the existing console consumer could be modified to do this?
>
> -Jay
>
> On Tue, Oct 9, 2012 at 10:05 PM, howard chen <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>>
>> On Wed, Oct 10, 2012 at 12:20 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:
>> > I am not aware of anyone having open sourced what you describe. If anyone
>> > is interested one nice way to implement this would be to implement a
>> daemon
>> > that could interact with the syslogd protocol over a unix domain socket.
>> > Many scripting languages have good syslogd integration and this would
>> > probably be a good strategy for supporting logging from this kind of
>> > process.
>> >
>>
>> In the simplest way, able to tail a file would be enough for this purpose.
>>
>> Existing app keep writing to local log file, no any blocking and not
>> affected by network connection, and another process tail the file and
>> send the changes to broker.
>>
>> Scribe and Flume also have this build in, would be nice if Kafka also
>> support this out of the box.
>>
>> (From old posts it was said that Window does not have tail...well, I
>> think most people run on Linux anyway..)
>>
>> Thanks.
>>

Yeah, the console producer should be able to do this out of the box.

The only concern would be to make sure log rotation is handled properly,
but I'm sure that wouldn't be too hard...

--
Felix

On Wed, Oct 10, 2012 at 1:15 PM, Zsolt Dollenstein <[EMAIL PROTECTED]>wrote:

> You mean the producer?
>
> On Wed, Oct 10, 2012 at 5:23 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > Yes, perhaps the existing console consumer could be modified to do this?
> >
> > -Jay
> >
> > On Tue, Oct 9, 2012 at 10:05 PM, howard chen <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> On Wed, Oct 10, 2012 at 12:20 AM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> >> > I am not aware of anyone having open sourced what you describe. If
> anyone
> >> > is interested one nice way to implement this would be to implement a
> >> daemon
> >> > that could interact with the syslogd protocol over a unix domain
> socket.
> >> > Many scripting languages have good syslogd integration and this would
> >> > probably be a good strategy for supporting logging from this kind of
> >> > process.
> >> >
> >>
> >> In the simplest way, able to tail a file would be enough for this
> purpose.
> >>
> >> Existing app keep writing to local log file, no any blocking and not
> >> affected by network connection, and another process tail the file and
> >> send the changes to broker.
> >>
> >> Scribe and Flume also have this build in, would be nice if Kafka also
> >> support this out of the box.
> >>
> >> (From old posts it was said that Window does not have tail...well, I
> >> think most people run on Linux anyway..)
> >>
> >> Thanks.
> >>
>

please excuse my brevity on this mobile device
On Oct 10, 2012 10:16 AM, "Zsolt Dollenstein" <[EMAIL PROTECTED]> wrote:

> You mean the producer?
>
> On Wed, Oct 10, 2012 at 5:23 PM, Jay Kreps <[EMAIL PROTECTED]> wrote:
> > Yes, perhaps the existing console consumer could be modified to do this?
> >
> > -Jay
> >
> > On Tue, Oct 9, 2012 at 10:05 PM, howard chen <[EMAIL PROTECTED]> wrote:
> >
> >> Hi,
> >>
> >> On Wed, Oct 10, 2012 at 12:20 AM, Jay Kreps <[EMAIL PROTECTED]>
> wrote:
> >> > I am not aware of anyone having open sourced what you describe. If
> anyone
> >> > is interested one nice way to implement this would be to implement a
> >> daemon
> >> > that could interact with the syslogd protocol over a unix domain
> socket.
> >> > Many scripting languages have good syslogd integration and this would
> >> > probably be a good strategy for supporting logging from this kind of
> >> > process.
> >> >
> >>
> >> In the simplest way, able to tail a file would be enough for this
> purpose.
> >>
> >> Existing app keep writing to local log file, no any blocking and not
> >> affected by network connection, and another process tail the file and
> >> send the changes to broker.
> >>
> >> Scribe and Flume also have this build in, would be nice if Kafka also
> >> support this out of the box.
> >>
> >> (From old posts it was said that Window does not have tail...well, I
> >> think most people run on Linux anyway..)
> >>
> >> Thanks.
> >>
>