I'm working on offset management for my Python client (non-ZK). I'm having trouble seeing how you would keep track of the message offset when using compression. As I understand it, when you use compression, you concatenate many messages together and then compress the resulting encoded MessageSet. How could you possibly keep track of the message offsets when doing this? As best as I can figure, you can only determine commit offsets for top-level messages - not nested ones (as you get with compression).
Jay Kreps 2012-11-19, 20:59