Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> producer exceptions when broker dies


Copy link to this message
-
Re: producer exceptions when broker dies
Guozhang,

It turns out this is not entirely true, you do need request.required.acks =
-1 (and yes, you need to retry if failure) in order have guaranteed
delivery.

I discovered this, when doing tests with rolling restarts (and hard
restarts) of the kafka servers.  If the server goes down, e.g. if there's
any change in the ISR leader for a partition, then the only way you can be
sure what you produced will be available on a newly elected leader is to
use -1.

Jason
On Sat, Oct 26, 2013 at 12:11 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Jason,
>
> Setting request.required.acks=-1 is orthogonal to the 'at least once'
> guarantee, it only relates to the latency/replication trade-off. For
> example, even if you set request.required.acks to 1, and as long as you
> retry on all non-fatal exceptions you have the "at least once" guarantee;
> and even if you set request.required.acks to -1 and you do not retry, you
> will not get "at least once".
>
> As you said, setting request.required.acks=-1 only means that when a
> success response has been returned you know that the message has been
> received by all ISR replicas, but this has nothing to do with "at least
> once" guarantee.
>
> Guozhang
>
>
> On Fri, Oct 25, 2013 at 8:55 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Just to clarify, I think in order to get 'at least once' guarantees, you
> > must produce messages with 'request.required.acks=-1'.  Otherwise, you
> > can't be 100% sure the message was received by all ISR replicas.
> >
> >
> > On Fri, Oct 25, 2013 at 9:56 PM, Kane Kane <[EMAIL PROTECTED]>
> wrote:
> >
> > > Thanks Guozhang, it makes sense if it's by design. Just wanted to
> ensure
> > > i'm not doing something wrong.
> > >
> > >
> > > On Fri, Oct 25, 2013 at 5:57 PM, Guozhang Wang <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > As we have said, the timeout exception does not actually mean the
> > message
> > > > is not committed to the broker. When message.send.max.retries is 0
> > Kafka
> > > > does guarantee "at-most-once" which means that you will not have
> > > > duplicates, but not means that all your exceptions can be treated as
> > > > "message not delivered". In your case, 1480 - 1450 = 30 messages are
> > the
> > > > ones that are actually sent, not the ones that are duplicates.
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Fri, Oct 25, 2013 at 5:00 PM, Kane Kane <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > There are a lot of exceptions, I will try to pick an example of
> each:
> > > > > ERROR async.DefaultEventHandler - Failed to send requests for
> topics
> > > > > benchmark with correlation ids in [879,881]
> > > > > WARN  async.DefaultEventHandler - Produce request with correlation
> id
> > > 874
> > > > > failed due to [benchmark,43]: kafka.common.RequestTimedOutException
> > > > > WARN  client.ClientUtils$ - Fetching topic metadata with
> correlation
> > id
> > > > 876
> > > > > for topics [Set(benchmark)] from broker
> > > > [id:2,host:10.80.42.156,port:9092]
> > > > > failed
> > > > > ERROR producer.SyncProducer - Producer connection to
> > > > > 10.80.42.156:9092unsuccessful
> > > > > kafka.common.FailedToSendMessageException: Failed to send messages
> > > after
> > > > 0
> > > > > tries.
> > > > > WARN  async.DefaultEventHandler - Failed to send producer request
> > with
> > > > > correlation id 270 to broker 0 with data for partitions
> > [benchmark,42]
> > > > >
> > > > > I think these are all types of exceptions i see there.
> > > > > Thanks.
> > > > >
> > > > >
> > > > > On Fri, Oct 25, 2013 at 2:45 PM, Guozhang Wang <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Kane,
> > > > > >
> > > > > > If you set message.send.max.retries to 0 it should be
> at-most-once,
> > > > and I
> > > > > > saw your props have the right config. What are the exceptions you
> > got
> > > > > from
> > > > > > the send() call?
> > > > > >
> > > > > > Guozhang
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 25, 2013 at 12:54 PM, Steve Morin <

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB