Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka

Copy link to this message
Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka

This is an automatically generated e-mail. To reply, visit:
My views (which hardly matter !!) about the points in 'Questions/comments for discussion' section:

- Change in partition assignment would need sync of offsets across brokers and subsequent bootstrap. Would be better to address in a separate jira.
- +1 for single error code for all offsets. There must be versioning for OffsetCommitResponse as in future there might be need to have separate error codes.
- It would be good to have separate error codes so that clients are aware of what went wrong.
- Lazy offset loading:
  - "clear out the offset cache on becoming a follower" : The cleanup thread would eventually clean this stuff after the retention period. As per on-paper-math, having extra offsets won't eat up much space in the offsets cache. There must be a mechanism (Map[partition -> lastKnownLeaderEpoch] ??) to figure out that those offsets in the cache are stale and a bootstrap is needed for them. As a downside, the old offsets would sit for long in the old generation space in heap eating up few MBs worth of space.
  - How would offset manager populate Map[partition -> lastKnownLeaderEpoch] ? Its clear that it can't do that all by itself thus implying some coupling with other module.
  - DurableLog: I liked the concept. This might touch a lot of places in the codebase and would be better to address in a separate jira.

    1. This could be made a 'val' instead of 'var'
    2. Are entries ever cleared off this pool ? I see that it will keep growing after rebalances.


    There are two different names used possibly for the same thing: offset coordinator and offset manager. Might be confusing to someone new top this feature.


    I am wondering why is an infinite loop needed inside offset commit. A finite retry counter might help for getting few reattempts.
- Tejas Patil
On Feb. 12, 2014, 7:50 p.m., Joel Koshy wrote: