Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka


Copy link to this message
-
Re: Review Request 18022: KAFKA-1012: In-built offset management in Kafka

This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18022/#review34615
Although I am NOT knowledgeable enough to review Joel's code, did a partial walk-over. Would add more comments after putting more thought.
core/src/main/scala/kafka/server/KafkaApis.scala
<https://reviews.apache.org/r/18022/#comment64790>

    For small batch of offset message(s), compression would increase the message size. For a setup, where large % of consumers read from few topic-partitions, this would be bad.

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64793>

    What was the reason behind choosing 10000 ? If there was any, it would be awesome to have it in comment.

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64794>

    scheduled task needs a better name

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64796>

    Is there any naming convention across the codebase for naming the start time of some operation ?

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64797>

    This is a corner case:
    - The cleaner thread populates the offset entry to be cleaned (say K, V1)
    - Broker's request handler thread does an offset commit with (K, V2)
    - Cleaner thread removes K from the offset cache.
    - (K, V2) is lost

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64798>

    This is a corner case:
    - The cleaner thread populates the tombstone message (K, null)
    - Broker receives a offset commit message and appends it to the log (K, V)
    - Cleaner thread appends the tombstone message to the log (K, null)
    - (K, V) is lost

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64799>

    A rollback would be really hard at this point after the message was written to logs.
    Shouldn't the consumer be notified about this ?

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64795>

    contains unsafe check-then-act

core/src/main/scala/kafka/server/OffsetManager.scala
<https://reviews.apache.org/r/18022/#comment64792>

    currently not used anywhere in the code
- Tejas Patil
On Feb. 12, 2014, 7:50 p.m., Joel Koshy wrote: