We are using 0.72 version of Kafka on Windows. I am wondering what is the right way to fetch data and keep track of offset in a partition. For example, I am currently assuming the first message the producer sent to the broker is at offset 0. So far it seems working. Is this correct assumption?
Let' say 2 days later, the first 100 messages on the broker is discarded because it passed retention.hours set in the config file. Now what is the offset I should use to retrieve the first message in the partition? And let's also say the offset I had for the 80th message is now not valid. What is the right way to get the correct offset to fetch in the consumer?
What is the purpose of the api for getting a list of valid offsets for all segments in a partition?
Since offsets in Kafka 0.7x are just byte counts, you cannot know the number of messages remaining to be processed (subtract your consumers offsets from each partitions end offset). However, you can know the number of bytes remaining. Knowing the average message size, you can use that to make a rough guess as to how many messages are remaining. On Mon, Jan 28, 2013 at 8:03 PM, S Ahmed <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext