Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Producer.send questions


Copy link to this message
-
Re: Producer.send questions
>> So, can QueueFullException occur in either sync or async mode (or just async
mode)?

QueueFullException can only occur in async mode since there is no queue in
sync mode.

>> If there's a MessageSizeTooLargeException, is there any visibility of
thisto the caller?

The kafka producer should not retry on unrecoverable exceptions. I'm not
sure we do this
correctly right now. In any case, this is a bug that should be fixed inside
the kafka producer.

>> I gathered from one of your previous responses, that a MessageSizeTooLargeException
can be rectified with a smaller batch size.
 If so, does that imply that the message size limit is measured on the
broker by the cumulative size of the batch, and not of any one message?

That's right. The broker does the message size check on the compressed
message. The size of the compressed message
is proportional to the batch size. Hence, reducing the batch size on a
retry might make sense here, but currently the
producer doesn't do this.

>> If I want to implement guaranteed delivery semantics, using the new request.required.acks
configuration, I need to expose retry logic beyond
that built into the producer?

The kafka producer must handle recoverable exceptions with a configurable
number of retries and must not retry
on unrecoverable exceptions. So ideally you shouldn't have to write your
own batching and retry logic.

Thanks,
Neha
On Sat, Aug 24, 2013 at 9:54 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Jun,
>
> Thanks, this is helpful.
>
> So, can QueueFullException occur in either sync or async mode (or just
> async mode)?
>
> If there's a MessageSizeTooLargeException, is there any visibility of this
> to the caller?  Or will it just be a FailedToSendMessageException.  I
> gathered from one of your previous responses, that a
> MessageSizeTooLargeException can be rectified with a smaller batch size.
>  If so, does that imply that the message size limit is measured on the
> broker by the cumulative size of the batch, and not of any one message?
>  (makes sense if the broker doesn't unwrap a batch of messages before
> storing on the server).
>
> If I want to implement guaranteed delivery semantics, using the new
> request.required.acks configuration, I need to expose retry logic beyond
> that built into the producer?  And to do this, I need to indicate to the
> caller whether it's possible to retry, or whether it will be fruitless.  I
> suppose allowing message.max.send.retries to allow infinite retries (e.g.
> by setting it to -1) might be useful.  But optionally, I'd like the caller
> to be able to handle this retry logic itself.
>
> Jason
>
>
> On Sat, Aug 24, 2013 at 8:22 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > You don't need to restart the producer. The producer currently handles
> all
> > error/exceptions by refreshing the metadata and retrying. If it fails all
> > retries, it throws a FailedToSendMessageException to the caller (in sync
> > mode). The original cause is not included in this exception. We have
> > thought about being a bit smarter in the producer retry logic such that
> it
> > only retries on recoverable errors and could implement this at some
> point.
> > Other than FailedToSendMessageException, the producer can also throw
> > QueueFullException.
> > This is an indication that the producer is sending data at a rate faster
> > than the broker can handle. This may or may not be recoverable since it
> > depends on the load.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Sat, Aug 24, 2013 at 1:44 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Jun,
> > >
> > > There are several others I've seen that I would have thought would be
> > > retryable (possibly after an exponential backoff delay).  I'm curious
> > > about:
> > >
> > > BrokerNotAvailableException
> > > FailedToSendMessageException
> > > QueueFullException (happens if producerType is 'async')
> > > KafkaException (this seems to wrap lots of base conditions, does one
> have
> > > to sort through the different wrapped exception types?)
So, can QueueFullException occur in either sync or async mode (or just
async mode)?

If there's a MessageSizeTooLargeException, is there any visibility of this
to the caller?  Or will it just be a FailedToSendMessageException.  I
gathered from one of your previous responses, that a
MessageSizeTooLargeException can be rectified with a smaller batch size.
 If so, does that imply that the message size limit is measured on the
broker by the cumulative size of the batch, and not of any one message?
 (makes sense if the broker doesn't unwrap a batch of messages before
storing on the server).

If I want to implement guaranteed delivery semantics, using the new
request.required.acks configuration, I need to expose retry logic beyond
that built into the producer?