Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - dumb question about offsets


Copy link to this message
-
Re: dumb question about offsets
Neha Narkhede 2012-11-22, 18:30
That's right.

On Thu, Nov 22, 2012 at 7:37 AM, Paul Garner
<[EMAIL PROTECTED]> wrote:
> sorry, just to be clear... in both 0.7 and 0.8 the offset keeps increasing,
> just in 0.8 it's no longer a byte position?
>
> On Thu, Nov 22, 2012 at 3:33 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> Yes, in Kafka 0.7, the offset is the byte position of the message in the
>> log for the topic partition. In Kafka 0.8, each message is assigned a
>> monotonically increasing, contiguous sequence number per partition,
>> starting with 1. So each message is addressable using this sequence number
>> instead of the byte position.
>>
>> Also, the offset keeps increasing over the lifetime of a cluster, even if
>> Kafka deletes older log segments.
>>
>> Thanks,
>> Neha
>>
>> On Thursday, November 22, 2012, Paul Garner wrote:
>>
>> > from what I read, the message offset is the byte position of the message
>> in
>> > the log file that Kafka writes to
>> >
>> > the logs are rotated and eventually deleted by Kafka
>> >
>> > ...does this mean the message offset periodically goes back to start at
>> > zero again? or the offset keeps increasing for the life of the cluster as
>> > if it was a single big file back to the beginning of time?
>> >
>>