Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Kafka and no guarantee that every published message is actually received by the broker


Copy link to this message
-
Re: Kafka and no guarantee that every published message is actually received by the broker
Neha Narkhede 2014-02-04, 17:30
We have added intra cluster replication to address the durability issue in
Kafka 0.8. You can read the latest on the design and guarantees here -
http://kafka.apache.org/documentation.html#semantics

Thanks
Neha
On Tue, Feb 4, 2014 at 8:52 AM, Christian Schuhegger <
[EMAIL PROTECTED]> wrote:

> Hello all,
>
> I was reading in the following paper:
>
> http://research.microsoft.com/en-us/um/people/srikanth/
> netdb11/netdb11papers/netdb11-final12.pdf
>
> the following paragraph:
>
> -- snip start --
> There are a few reasons why Kafka performed much better. First,
> the Kafka producer currently doesn't wait for acknowledgements
> from the broker and sends messages as faster as the broker can
> handle. This significantly increased the throughput of the
> publisher. With a batch size of 50, a single Kafka producer almost
> saturated the 1Gb link between the producer and the broker. This
> is a valid optimization for the log aggregation case, as data must
> be sent asynchronously to avoid introducing any latency into the
> live serving of traffic. We note that without acknowledging the
> producer, there is no guarantee that every published message is
> actually received by the broker. For many types of log data, it is
> desirable to trade durability for throughput, as long as the number
> of dropped messages is relatively small. However, we do plan to
> address the durability issue for more critical data in the future.
> -- snip end --
>
> And I was wondering if this is still true or if the plans for the future
> as described above to address the durability issue for more critical data
> were realized?
>
> Many thanks,
> --
> Christian Schuhegger
>
>
>