Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Keeping track of offsets

Copy link to this message
Keeping track of offsets
David Arthur 2012-11-19, 20:45
I'm working on offset management for my Python client (non-ZK). I'm having trouble seeing how you would keep track of the message offset when using compression. As I understand it, when you use compression, you concatenate many messages together and then compress the resulting encoded MessageSet. How could you possibly keep track of the message offsets when doing this? As best as I can figure, you can only determine commit offsets for top-level messages - not nested ones (as you get with compression).