Kafka, mail # user - Re: are kafka consumer apps guaranteed to see msgs at least once? - 2013-11-23, 11:42
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: are kafka consumer apps guaranteed to see msgs at least once?
Interesting, but I think you might run into problems when partitions get
rebalanced between processes (not just between threads in the same process).

So, you'd need to coordinate your issue of not committing backward, between
threads running on different processes.  I'm trying to think now, what
might be the worst case if you commit an offset lower than one that was
already committed by another thread/process.  I guess the danger is only in
truly unnecessary duplicate processing?  Or is there an actual danger that
the system might flail and never move forward!?

Are you attempting to add commitOffsetForPartition(partition, offset)
yourself, as a mod to the ZookeeperConsumerConnector?  One thing that's
missing too, is some sort of call back by the ConsumerConnector to notify
when a rebalance is happening.  This would be useful for trying to make
consumer apps a bit more proactive in that case, if not using the default
autocommit.

Interesting stuff....

Jason
On Sat, Nov 23, 2013 at 12:55 AM, Imran Rashid <[EMAIL PROTECTED]> wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB