Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Producer.send questions


Copy link to this message
-
Re: Producer.send questions
Neha Narkhede 2013-08-25, 03:26
>> Ok, but perhaps the producer will handle something like this in the
future?

Yes, I think we need a JIRA for this.

>> UnretryableFailedToSendMessageException (wraps the root cause)
NoMoreRetriesFailedToSendMessageException (wraps the root cause, from the
final attempt)

Something like this makes sense. Would you mind creating a JIRA for this so
we can
discuss a solution there ?

Thanks,
Neha
On Sat, Aug 24, 2013 at 10:41 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Thanks Neha,
>
> On Sat, Aug 24, 2013 at 10:06 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> > >> I gathered from one of your previous responses, that a
> > MessageSizeTooLargeException
> > can be rectified with a smaller batch size.
> >  If so, does that imply that the message size limit is measured on the
> > broker by the cumulative size of the batch, and not of any one message?
> >
> > That's right. The broker does the message size check on the compressed
> > message. The size of the compressed message
> > is proportional to the batch size. Hence, reducing the batch size on a
> > retry might make sense here, but currently the
> > producer doesn't do this.
> >
>
> Ok, but perhaps the producer will handle something like this in the future?
>
> >
> > >> If I want to implement guaranteed delivery semantics, using the new
> > request.required.acks
> > configuration, I need to expose retry logic beyond
> > that built into the producer?
> >
> > The kafka producer must handle recoverable exceptions with a configurable
> > number of retries and must not retry
> > on unrecoverable exceptions. So ideally you shouldn't have to write your
> > own batching and retry logic.
> >
>
> So, it seems there might be a bit of a gray area.  There is a configurable
> retry count, which we can increase perhaps to gain confidence that anything
> recoverable has been sent.  But, since this retry count is finite, there's
> no way to know for sure that it won't succeed if it were retried just one
> more time.  So, it is then difficult to conclude that  if Producer.send
> throws a FailedToSendMessageException, the message shouldn't be retried.
>
> Perhaps it would be useful to define different exception types, so that a
> caller can have clearer semantics:
>
> UnretryableFailedToSendMessageException (wraps the root cause)
> NoMoreRetriesFailedToSendMessageException (wraps the root cause, from the
> final attempt)
>
> Probably shorter names are possible here!  Perhaps these could be
> subclasses of FailedToSendMessageException.  Alternately,
> FailedToSendMessageException could include information, such as the number
> of retries attempted, and a flag indicating whether it's possible to retry
> the message.
>
> Jason
>
>
> >
> >
> >
> > On Sat, Aug 24, 2013 at 9:54 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Jun,
> > >
> > > Thanks, this is helpful.
> > >
> > > So, can QueueFullException occur in either sync or async mode (or just
> > > async mode)?
> > >
> > > If there's a MessageSizeTooLargeException, is there any visibility of
> > this
> > > to the caller?  Or will it just be a FailedToSendMessageException.  I
> > > gathered from one of your previous responses, that a
> > > MessageSizeTooLargeException can be rectified with a smaller batch
> size.
> > >  If so, does that imply that the message size limit is measured on the
> > > broker by the cumulative size of the batch, and not of any one message?
> > >  (makes sense if the broker doesn't unwrap a batch of messages before
> > > storing on the server).
> > >
> > > If I want to implement guaranteed delivery semantics, using the new
> > > request.required.acks configuration, I need to expose retry logic
> beyond
> > > that built into the producer?  And to do this, I need to indicate to
> the
> > > caller whether it's possible to retry, or whether it will be fruitless.
> >  I
> > > suppose allowing message.max.send.retries to allow infinite retries
> (e.g.
> > > by setting it to -1) might be useful.  But optionally, I'd like the