Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - kafka-49


Copy link to this message
-
Re: kafka-49
Jeffrey Damick 2011-07-21, 14:03
But if there are no replicas, a produce with an ACK requested would at least
force a flush to disk I assume?  That would be enough for me right now.
And I agree with Jay's comment, the tradeoff of durability vs. throughput
could be chosen by the producer, either with a different message type or
flag in the request.

As for the offset of the message, I'd really like to be able to know what it
is after I produce so that it can be logged and monitored.  Then if needed
something or someone can quickly grab the latest message produced on a topic
for metrics, monitoring, and debugging if something goes awry, otherwise I
need to have the producer also consume the topic to figure it out.. (the
data I'm transporting is a little more important than logs)

So back to the ACK'ing, I'd willing to submit a patch for at least the
optional flag or new message type to force a flush to disk if it would
help..

thanks,
-jeff
On Wed, Jul 20, 2011 at 8:51 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jeff,
>
> I was thinking of making the ACK mandatory for the producer. The ACK can be
> sent when the message either hits 1 replicas or multiple replicas,
> depending
> on the setting.
>
> Having the ACK include the starting offset of the message seems reasonable.
> It will be a bit complicated for multisend since multiple offsets have to
> be
> returned. What do you need the offset for?
>
> Thanks,
>
> Jun
>
> On Wed, Jul 20, 2011 at 12:47 PM, Jeffrey Damick <[EMAIL PROTECTED]
> >wrote:
>
> > Is there any current thought around KAFKA-49, for acknowledgement of
> > producers?
> > Will this be optional, a new message type(s)?
> > Will the ack be synchronous or asynchronous or depending on request type?
> >
> > It would be fantastic if the ack contained the starting offset of the
> > message published, and not just the ending.
> >
> > This is quickly becoming an issue for us, so we may be able to provide
> some
> > help in this area..
> >
> >
> > thanks
> > -jeff
> >
>