Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - [0.8] Autocommit bug? Or am I doing it wrong? :)


Copy link to this message
-
Re: [0.8] Autocommit bug? Or am I doing it wrong? :)
Jun Rao 2013-05-24, 16:33
This turns out to be a bug. It seems that we always call commitOffsets() in
ZookeeperConsumerConnector.closeFetchersForQueues() (which is called during
rebalance), whether auto commit is enabled or not. Could you file a jira?

Thanks,

Jun
On Fri, May 24, 2013 at 8:06 AM, Hargett, Phil <
[EMAIL PROTECTED]> wrote:

> In one of our applications using Kafka, we are using the high-level
> consumer to pull messages from our topic.
>
> Because we pull messages from topics in discrete units (e.g., an hour's
> worth of messages), we want to control explicitly when offsets are
> committed.
>
> Even though "auto.commit.enable" is set to false, during consumer group
> rebalancing, offsets are committed anyway, regardless of the setting of
> this flag.
>
> Is this a bug? Or just a known gap in offset management? I do see plenty
> of notes on the wiki suggesting future releases may enable applications
> using the high-level consumer to have more fine-grained control over offset
> management.
>
> I also fully realize that different applications have different needs, and
> meeting all of them with a clean API can be challenging.
>
> In the case of this application, the high-level consumer solves the
> problem of locating the correct in a cluster for a given topic, so there
> are advantages to using it, even if we are not using it to balance fetch
> load across multiple consumers. We ideally have only 1 consumer active per
> consumer group, and can tolerate some duplicate messages. But, the consumer
> groups make it easy for 1 consumer to recover at the correct starting
> point, should the prior consumer in the group have failed before doing a
> commit.
>
> Thoughts?