This turns out to be a bug. It seems that we always call commitOffsets() in
ZookeeperConsumerConnector.closeFetchersForQueues() (which is called during
rebalance), whether auto commit is enabled or not. Could you file a jira?
On Fri, May 24, 2013 at 8:06 AM, Hargett, Phil <
[EMAIL PROTECTED]> wrote:
> In one of our applications using Kafka, we are using the high-level
> consumer to pull messages from our topic.
> Because we pull messages from topics in discrete units (e.g., an hour's
> worth of messages), we want to control explicitly when offsets are
> Even though "auto.commit.enable" is set to false, during consumer group
> rebalancing, offsets are committed anyway, regardless of the setting of
> this flag.
> Is this a bug? Or just a known gap in offset management? I do see plenty
> of notes on the wiki suggesting future releases may enable applications
> using the high-level consumer to have more fine-grained control over offset
> I also fully realize that different applications have different needs, and
> meeting all of them with a clean API can be challenging.
> In the case of this application, the high-level consumer solves the
> problem of locating the correct in a cluster for a given topic, so there
> are advantages to using it, even if we are not using it to balance fetch
> load across multiple consumers. We ideally have only 1 consumer active per
> consumer group, and can tolerate some duplicate messages. But, the consumer
> groups make it easy for 1 consumer to recover at the correct starting
> point, should the prior consumer in the group have failed before doing a