Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> log file flushing, flushall on clean shutdown...

Copy link to this message
Re: log file flushing, flushall on clean shutdown...

How do you know the server had written the lost data to its log ? In Kafka
0.7, data could be lost from the producer's or server's socket buffer. You
can verify this by running DumpLogSegments before and after shutdown.


On Thursday, March 28, 2013, SuoNayi wrote:

> Hi, the mail list has been moved to [EMAIL PROTECTED] <javascript:;>long time ago.
> You may resubscribe the list or the list may be removed in future.
> At 2013-03-28 15:39:07,"Jason Rosenberg" <[EMAIL PROTECTED] <javascript:;>>
> wrote:
> >Hi,
> >
> >We are managing our kafka clusters by doing rolling restarts (e.g. we
> >cleanly shutdown and restart each broker one at a time).
> >
> >I'm working through an issue, whereby we are losing messages sent to a
> >kafka broker, right before the broker is shutdown cleanly.
> >
> >I'm still using 0.7.2, so not sure if this is also the behavior in 0.8.
> >
> >I am trying to cleanly shutting down the kafka server by calling
> >kafka.server.KafkaServer.shutdown().  I then wait for the shutdown to
> >complete, by calling awaitShutdown().
> >
> >However, it looks like this does not attempt to flush all logs before
> >quitting, as I had assumed was the case.  Is there a way to make sure
> >this happens before the server stops?
> >
> >Jason