Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Offset question


Copy link to this message
-
Re: Offset question
Offsets always begin at 0 for each partition and increase sequentially from
there. Offsets aren't unique within a topic. As old data is discarded the
first retained offset will not remain 0. The behavior of what is retained
is controlled by your retention settings.

In trunk there is a feature that allows removing messages from the log (
https://cwiki.apache.org/confluence/display/KAFKA/Log+Compaction). This
would leave holes between offsets for topics where you enabled compaction.

-Jay
On Mon, Oct 7, 2013 at 12:45 PM, Francis Dallaire <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have question regarding the offset in kafka (0.8). I've gone through the
> documentation and did some tests, but I want to make sure I'm on the right
> track.
>
>
> *         Are the offsets guaranteed to be sequential in a partition ?
>
> o   Can it contain holes ?
>
> *         How offsets are distributed amongst partitions ?
>
> o   Unique inside a partition, not inside a topic ?
>
> o   What determines where it starts for each partition ?
>
> If you have a link to some documentation, that could be useful.
>
> Thanks,
>
> Francis
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB