Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: commitOffsets by partition 0.8-beta


Copy link to this message
-
Re: commitOffsets by partition 0.8-beta
Hey guys, I am dealing with a similar problem and hoping a similar solution can help me out. Looking for some feedback on this problem and potential solution:

So I am reading messages from a topic, then doing some synchronous processing in the thread handling the consumer iterator, THEN issuing an asynchronous write operation to our persistent data store. (it's asynchronous for performance reasons, if we flushed that through on every write it would be very slow.) So the current message offset of the consumer thread won't necessarily correspond to what has been flushed to the persistent data store. I'm thinking I can keep track of the earliest outstanding message offset per partition we've finished flushing, and then in a callback on every flush, also commit that offset for each partition to Zookeeper.

Thanks in advance,
On Apr 23, 2014, at 12:01 PM, Seshadri, Balaji <[EMAIL PROTECTED]> wrote:

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB