Hi,

We are currently using the Log4j appender to write to the Kafka (on
machine on another datacenter), but problem occur when the network is
interrupted as the message is stored in memory.

Are there any local forwarder/client or similar solutions that can be
installed on the localhost and take care of the message forwarding?

Thanks.

 

Ryan,

Have you tried configuring the num.retries option on the log4j producer ?
When there is a temporary network glitch, it will retry sending the
messages instead of losing them.

Thanks,
Neha

On Thursday, March 14, 2013, Ryan Chan wrote:

> Hi,
>
> We are currently using the Log4j appender to write to the Kafka (on
> machine on another datacenter), but problem occur when the network is
> interrupted as the message is stored in memory.
>
> Are there any local forwarder/client or similar solutions that can be
> installed on the localhost and take care of the message forwarding?
>
> Thanks.
>

 

Hi,

On Fri, Mar 15, 2013 at 1:18 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
> Ryan,
>
> Have you tried configuring the num.retries option on the log4j producer ?
> When there is a temporary network glitch, it will retry sending the
> messages instead of losing them.
>

Memory is limited and this is our concern. (We are sending in a quite high rate)
We want to search for a more flexible solution like sync to local disk
and have other process do the retry.
Thanks.

 

https://issues.apache.org/jira/browse/KAFKA-156 is filed for that feature.

Thanks,
Neha
On Fri, Mar 15, 2013 at 12:55 AM, Ryan Chan <[EMAIL PROTECTED]> wrote:

> Hi,
>
> On Fri, Mar 15, 2013 at 1:18 PM, Neha Narkhede <[EMAIL PROTECTED]>
> wrote:
> > Ryan,
> >
> > Have you tried configuring the num.retries option on the log4j producer ?
> > When there is a temporary network glitch, it will retry sending the
> > messages instead of losing them.
> >
>
> Memory is limited and this is our concern. (We are sending in a quite high
> rate)
> We want to search for a more flexible solution like sync to local disk
> and have other process do the retry.
>
>
> Thanks.
>