I am new to Kafka and I have a question how Kafka handles scenarios where no consumer is available. Can I configure Kafka in such a way that the messages will be dropped after x seconds? Otherwise I would be afraid that the queues would overflow...
Message retention in Kafka is disconnected from message consumption. Messages are all persisted to disk and the queues do not need to fit in RAM unlike some other systems. There are configuration values that control maximum log size in terms of MB and the duration of retention which is typically in terms of days, weeks or months, though perhaps hours at very high volumes. While you certainly could configure Kafka so that it would run out of disk space this can be avoided by a combination of configuration changes and bigger or more cheap spinning disks, or distributing the data across more machines.
I hope this helps, though others likely have the configuration values at their fingertips.
Christian On Jun 27, 2014 1:09 AM, "Klaus Schaefers" <[EMAIL PROTECTED]> wrote:
You can control retention using "log.retention.hours", "log.retention.minutes" or "log.retention.bytes". On Fri, Jun 27, 2014 at 2:06 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext