Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - [jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit


Copy link to this message
-
[jira] [Commented] (KAFKA-987) Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
"Neha Narkhede 2013-07-24, 16:55

    [ https://issues.apache.org/jira/browse/KAFKA-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718567#comment-13718567 ]

Neha Narkhede commented on KAFKA-987:
-------------------------------------

Jun,

I think what you are suggesting makes sense on startup before the consumer has consumed any messages. However, since the offset map is a cache for what's in zookeeper, the safest way is to keep it in sync with the zookeeper data. Before the consumer can pull any data, it has to rebalance and while rebalancing we read the offsets from zk anyways. So I think it is correct to update the offset cache in addPartitionTopicInfo()

                
> Avoid checkpointing offsets in Kafka consumer that have not changed since the last commit
> -----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-987
>                 URL: https://issues.apache.org/jira/browse/KAFKA-987
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8
>            Reporter: Swapnil Ghike
>            Assignee: Swapnil Ghike
>              Labels: improvement
>             Fix For: 0.8
>
>         Attachments: kafka-987.patch, kafka-987-v2.patch
>
>
> We need to fix the Kafka zookeeper consumer to avoid checkpointing offsets that have not changed since the last offset commit. This will help reduce the write load on zookeeper.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira