On Thu, Nov 22, 2012 at 07:33:31AM -0800, Neha Narkhede wrote:
> Yes, in Kafka 0.7, the offset is the byte position of the message in the
> log for the topic partition. In Kafka 0.8, each message is assigned a
> monotonically increasing, contiguous sequence number per partition,
> starting with 1. So each message is addressable using this sequence number
> instead of the byte position.
This is interesting. We at Loggly liked the offset, and thought it was an elegant idea (as explained on the Kafka design page). Are you *replacing* the offset, or will the sequence number be another way to reference a message?
And why the change? Perhaps there is a JIRA ticket explaining it in more detail.
> Also, the offset keeps increasing over the lifetime of a cluster, even if
> Kafka deletes older log segments.
> On Thursday, November 22, 2012, Paul Garner wrote:
> > from what I read, the message offset is the byte position of the message in
> > the log file that Kafka writes to
> > the logs are rotated and eventually deleted by Kafka
> > ...does this mean the message offset periodically goes back to start at
> > zero again? or the offset keeps increasing for the life of the cluster as
> > if it was a single big file back to the beginning of time?
San Francisco, CA