-Re: processing a batch of messages in a "transaction"
Neha Narkhede 2013-11-20, 14:45
You can turn off automatic offset commit (auto.commit.enable=false) and use
the commitOffsets() API. Note that this API will commit offsets for all
partitions owned by the consumer.
On Nov 20, 2013 6:39 AM, "Imran Rashid" <[EMAIL PROTECTED]> wrote:
> I have an application which reads messages from a kafka queue, builds
> up a batch of messages, and then performs some action on that batch.
> So far I have just used the ConsumerGroup api. However, I realized
> there is a potential problem -- my app may die sometime in the middle
> of the batch, and then those messages could get completely lost.
> Kafka will think we've already read those messages, and so on restart
> it will skip to messages post batch.
> 1) Does the ConsumerGroup api have some way to allow you to control
> when the the position of the consumer-group is updated? I couldn't
> find anything relevant in ConsumerGroup, KafkaStream, or
> ConsumerIterator, but perhaps I overlooked something.
> 2) Do I need to switch to SimpleConsumer? It looks like that does not
> use zookeeper at all, I'd rather not have to manage all of the
> zookeeper stuff myself.
> This seems like a basic use case, so I'm probably missing something ...
> (I'm using 0.8.0-beta1)