Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Committing offsets


Copy link to this message
-
Committing offsets
Suppose I'm iterating through messages from a topic with 6 messages. I consume message three messages and then commit (or an auto-commit happens).

message 1
message 2
message 3
                      <--- commit
message 4
message 5
message 6

I'm wondering what offset is committed here. Is it the beginning of message 3 or the end of message 3?

I'm particularly curious about what happens if I'm consuming messages and a commit occurs, but then something goes wrong and I fail to process that message (whatever "process" means here). Ideally, if I commit in the above scenario and then my consumer dies, the consumers will rebalance and someone else will pick up message 3. Is this what happens?

This also leads me to ponder about corrupt messages and retries and skipping "bad" messages...

Cheers,
David
+
Jun Rao 2012-11-08, 15:31
+
David Arthur 2012-11-08, 16:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB