Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Producer.send questions


Copy link to this message
-
Re: Producer.send questions
Actually we do have a JIRA tracking this issue:

https://issues.apache.org/jira/browse/KAFKA-998

And BTW, any review comments are welcome :)

Guozhang
On Sat, Aug 24, 2013 at 8:25 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> Ok, but perhaps the producer will handle something like this in the
> future?
>
> Yes, I think we need a JIRA for this.
>
> >> UnretryableFailedToSendMessageException (wraps the root cause)
> NoMoreRetriesFailedToSendMessageException (wraps the root cause, from the
> final attempt)
>
> Something like this makes sense. Would you mind creating a JIRA for this so
> we can
> discuss a solution there ?
>
> Thanks,
> Neha
>
>
> On Sat, Aug 24, 2013 at 10:41 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Thanks Neha,
> >
> > On Sat, Aug 24, 2013 at 10:06 AM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> > >
> > > >> I gathered from one of your previous responses, that a
> > > MessageSizeTooLargeException
> > > can be rectified with a smaller batch size.
> > >  If so, does that imply that the message size limit is measured on the
> > > broker by the cumulative size of the batch, and not of any one message?
> > >
> > > That's right. The broker does the message size check on the compressed
> > > message. The size of the compressed message
> > > is proportional to the batch size. Hence, reducing the batch size on a
> > > retry might make sense here, but currently the
> > > producer doesn't do this.
> > >
> >
> > Ok, but perhaps the producer will handle something like this in the
> future?
> >
> > >
> > > >> If I want to implement guaranteed delivery semantics, using the new
> > > request.required.acks
> > > configuration, I need to expose retry logic beyond
> > > that built into the producer?
> > >
> > > The kafka producer must handle recoverable exceptions with a
> configurable
> > > number of retries and must not retry
> > > on unrecoverable exceptions. So ideally you shouldn't have to write
> your
> > > own batching and retry logic.
> > >
> >
> > So, it seems there might be a bit of a gray area.  There is a
> configurable
> > retry count, which we can increase perhaps to gain confidence that
> anything
> > recoverable has been sent.  But, since this retry count is finite,
> there's
> > no way to know for sure that it won't succeed if it were retried just one
> > more time.  So, it is then difficult to conclude that  if Producer.send
> > throws a FailedToSendMessageException, the message shouldn't be retried.
> >
> > Perhaps it would be useful to define different exception types, so that a
> > caller can have clearer semantics:
> >
> > UnretryableFailedToSendMessageException (wraps the root cause)
> > NoMoreRetriesFailedToSendMessageException (wraps the root cause, from the
> > final attempt)
> >
> > Probably shorter names are possible here!  Perhaps these could be
> > subclasses of FailedToSendMessageException.  Alternately,
> > FailedToSendMessageException could include information, such as the
> number
> > of retries attempted, and a flag indicating whether it's possible to
> retry
> > the message.
> >
> > Jason
> >
> >
> > >
> > >
> > >
> > > On Sat, Aug 24, 2013 at 9:54 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Jun,
> > > >
> > > > Thanks, this is helpful.
> > > >
> > > > So, can QueueFullException occur in either sync or async mode (or
> just
> > > > async mode)?
> > > >
> > > > If there's a MessageSizeTooLargeException, is there any visibility of
> > > this
> > > > to the caller?  Or will it just be a FailedToSendMessageException.  I
> > > > gathered from one of your previous responses, that a
> > > > MessageSizeTooLargeException can be rectified with a smaller batch
> > size.
> > > >  If so, does that imply that the message size limit is measured on
> the
> > > > broker by the cumulative size of the batch, and not of any one
> message?
> > > >  (makes sense if the broker doesn't unwrap a batch of messages before
> > > > storing on the server).
> > > >
> > >

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB