Hey guys, I am dealing with a similar problem and hoping a similar solution can help me out. Looking for some feedback on this problem and potential solution:
So I am reading messages from a topic, then doing some synchronous processing in the thread handling the consumer iterator, THEN issuing an asynchronous write operation to our persistent data store. (it's asynchronous for performance reasons, if we flushed that through on every write it would be very slow.) So the current message offset of the consumer thread won't necessarily correspond to what has been flushed to the persistent data store. I'm thinking I can keep track of the earliest outstanding message offset per partition we've finished flushing, and then in a callback on every flush, also commit that offset for each partition to Zookeeper.
Thanks in advance,
On Apr 23, 2014, at 12:01 PM, Seshadri, Balaji <[EMAIL PROTECTED]> wrote: