Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Offset question

Copy link to this message
Re: Offset question
Offsets always begin at 0 for each partition and increase sequentially from
there. Offsets aren't unique within a topic. As old data is discarded the
first retained offset will not remain 0. The behavior of what is retained
is controlled by your retention settings.

In trunk there is a feature that allows removing messages from the log (
https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction). This
would leave holes between offsets for topics where you enabled compaction.

On Mon, Oct 7, 2013 at 12:45 PM, Francis Dallaire <

> Hi,
> I have question regarding the offset in kafka (0.8). I've gone through the
> documentation and did some tests, but I want to make sure I'm on the right
> track.
> *         Are the offsets guaranteed to be sequential in a partition ?
> o   Can it contain holes ?
> *         How offsets are distributed amongst partitions ?
> o   Unique inside a partition, not inside a topic ?
> o   What determines where it starts for each partition ?
> If you have a link to some documentation, that could be useful.
> Thanks,
> Francis