Kafka, mail # user - Re: Producer.send questions - 2013-08-24, 17:06
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: Producer.send questions
mode)?

QueueFullException can only occur in async mode since there is no queue in
sync mode.

thisto the caller?

The kafka producer should not retry on unrecoverable exceptions. I'm not
sure we do this
correctly right now. In any case, this is a bug that should be fixed inside
the kafka producer.

can be rectified with a smaller batch size.
 If so, does that imply that the message size limit is measured on the
broker by the cumulative size of the batch, and not of any one message?

That's right. The broker does the message size check on the compressed
message. The size of the compressed message
is proportional to the batch size. Hence, reducing the batch size on a
retry might make sense here, but currently the
producer doesn't do this.

configuration, I need to expose retry logic beyond
that built into the producer?

The kafka producer must handle recoverable exceptions with a configurable
number of retries and must not retry
on unrecoverable exceptions. So ideally you shouldn't have to write your
own batching and retry logic.

Thanks,
Neha
On Sat, Aug 24, 2013 at 9:54 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

So, can QueueFullException occur in either sync or async mode (or just
async mode)?

If there's a MessageSizeTooLargeException, is there any visibility of this
to the caller?  Or will it just be a FailedToSendMessageException.  I
gathered from one of your previous responses, that a
MessageSizeTooLargeException can be rectified with a smaller batch size.
 If so, does that imply that the message size limit is measured on the
broker by the cumulative size of the batch, and not of any one message?
 (makes sense if the broker doesn't unwrap a batch of messages before
storing on the server).

If I want to implement guaranteed delivery semantics, using the new
request.required.acks configuration, I need to expose retry logic beyond
that built into the producer?

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB