Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> producer exceptions when broker dies


Copy link to this message
-
Re: producer exceptions when broker dies
Guozhang,

It turns out this is not entirely true, you do need request.required.acks =
-1 (and yes, you need to retry if failure) in order have guaranteed
delivery.

I discovered this, when doing tests with rolling restarts (and hard
restarts) of the kafka servers.  If the server goes down, e.g. if there's
any change in the ISR leader for a partition, then the only way you can be
sure what you produced will be available on a newly elected leader is to
use -1.

Jason
On Sat, Oct 26, 2013 at 12:11 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Jason,
>
> Setting request.required.acks=-1 is orthogonal to the 'at least once'
> guarantee, it only relates to the latency/replication trade-off. For
> example, even if you set request.required.acks to 1, and as long as you
> retry on all non-fatal exceptions you have the "at least once" guarantee;
> and even if you set request.required.acks to -1 and you do not retry, you
> will not get "at least once".
>
> As you said, setting request.required.acks=-1 only means that when a
> success response has been returned you know that the message has been
> received by all ISR replicas, but this has nothing to do with "at least
> once" guarantee.
>
> Guozhang
>
>
> On Fri, Oct 25, 2013 at 8:55 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Just to clarify, I think in order to get 'at least once' guarantees, you
> > must produce messages with 'request.required.acks=-1'.  Otherwise, you
> > can't be 100% sure the message was received by all ISR replicas.
> >
> >
> > On Fri, Oct 25, 2013 at 9:56 PM, Kane Kane <[EMAIL PROTECTED]>
> wrote:
> >
> > > Thanks Guozhang, it makes sense if it's by design. Just wanted to
> ensure
> > > i'm not doing something wrong.
> > >
> > >
> > > On Fri, Oct 25, 2013 at 5:57 PM, Guozhang Wang <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > As we have said, the timeout exception does not actually mean the
> > message
> > > > is not committed to the broker. When message.send.max.retries is 0
> > Kafka
> > > > does guarantee "at-most-once" which means that you will not have
> > > > duplicates, but not means that all your exceptions can be treated as
> > > > "message not delivered". In your case, 1480 - 1450 = 30 messages are
> > the
> > > > ones that are actually sent, not the ones that are duplicates.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Fri, Oct 25, 2013 at 5:00 PM, Kane Kane <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > There are a lot of exceptions, I will try to pick an example of
> each:
> > > > > ERROR async.DefaultEventHandler - Failed to send requests for
> topics
> > > > > benchmark with correlation ids in [879,881]
> > > > > WARN  async.DefaultEventHandler - Produce request with correlation
> id
> > > 874
> > > > > failed due to [benchmark,43]: kafka.common.RequestTimedOutException
> > > > > WARN  client.ClientUtils$ - Fetching topic metadata with
> correlation
> > id
> > > > 876
> > > > > for topics [Set(benchmark)] from broker
> > > > [id:2,host:10.80.42.156,port:9092]
> > > > > failed
> > > > > ERROR producer.SyncProducer - Producer connection to
> > > > > 10.80.42.156:9092unsuccessful
> > > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > > after
> > > > 0
> > > > > tries.
> > > > > WARN  async.DefaultEventHandler - Failed to send producer request
> > with
> > > > > correlation id 270 to broker 0 with data for partitions
> > [benchmark,42]
> > > > >
> > > > > I think these are all types of exceptions i see there.
> > > > > Thanks.
> > > > >
> > > > >
> > > > > On Fri, Oct 25, 2013 at 2:45 PM, Guozhang Wang <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Kane,
> > > > > >
> > > > > > If you set message.send.max.retries to 0 it should be
> at-most-once,
> > > > and I
> > > > > > saw your props have the right config. What are the exceptions you
> > got
> > > > > from
> > > > > > the send() call?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 12:54 PM, Steve Morin <