Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> dumb question about offsets


Copy link to this message
-
Re: dumb question about offsets
Yes, in Kafka 0.7, the offset is the byte position of the message in the
log for the topic partition. In Kafka 0.8, each message is assigned a
monotonically increasing, contiguous sequence number per partition,
starting with 1. So each message is addressable using this sequence number
instead of the byte position.

Also, the offset keeps increasing over the lifetime of a cluster, even if
Kafka deletes older log segments.

Thanks,
Neha

On Thursday, November 22, 2012, Paul Garner wrote:

> from what I read, the message offset is the byte position of the message in
> the log file that Kafka writes to
>
> the logs are rotated and eventually deleted by Kafka
>
> ...does this mean the message offset periodically goes back to start at
> zero again? or the offset keeps increasing for the life of the cluster as
> if it was a single big file back to the beginning of time?
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB