Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> processing a batch of messages in a "transaction"

Copy link to this message
Re: processing a batch of messages in a "transaction"
You can turn off automatic offset commit (auto.commit.enable=false) and use
the commitOffsets() API. Note that this API will commit offsets for all
partitions owned by the consumer.

On Nov 20, 2013 6:39 AM, "Imran Rashid" <[EMAIL PROTECTED]> wrote:

> Hi,
> I have an application which reads messages from a kafka queue, builds
> up a batch of messages, and then performs some action on that batch.
> So far I have just used the ConsumerGroup api.  However, I realized
> there is a potential problem -- my app may die sometime in the middle
> of the batch, and then those messages could get completely lost.
> Kafka will think we've already read those messages, and so on restart
> it will skip to messages post batch.
> 1) Does the ConsumerGroup api have some way to allow you to control
> when the the position of the consumer-group is updated?  I couldn't
> find anything relevant in ConsumerGroup, KafkaStream, or
> ConsumerIterator, but perhaps I overlooked something.
> 2) Do I need to switch to SimpleConsumer?  It looks like that does not
> use zookeeper at all, I'd rather not have to manage all of the
> zookeeper stuff myself.
> This seems like a basic use case, so I'm probably missing something ...
> thanks,
> Imran
> (I'm using 0.8.0-beta1)