Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> processing a batch of messages in a "transaction"


Copy link to this message
-
Re: processing a batch of messages in a "transaction"
You can turn off automatic offset commit (auto.commit.enable=false) and use
the commitOffsets() API. Note that this API will commit offsets for all
partitions owned by the consumer.

Thanks,
Neha
On Nov 20, 2013 6:39 AM, "Imran Rashid" <[EMAIL PROTECTED]> wrote:

> Hi,
>
> I have an application which reads messages from a kafka queue, builds
> up a batch of messages, and then performs some action on that batch.
> So far I have just used the ConsumerGroup api.  However, I realized
> there is a potential problem -- my app may die sometime in the middle
> of the batch, and then those messages could get completely lost.
> Kafka will think we've already read those messages, and so on restart
> it will skip to messages post batch.
>
> 1) Does the ConsumerGroup api have some way to allow you to control
> when the the position of the consumer-group is updated?  I couldn't
> find anything relevant in ConsumerGroup, KafkaStream, or
> ConsumerIterator, but perhaps I overlooked something.
>
> 2) Do I need to switch to SimpleConsumer?  It looks like that does not
> use zookeeper at all, I'd rather not have to manage all of the
> zookeeper stuff myself.
>
> This seems like a basic use case, so I'm probably missing something ...
>
> thanks,
> Imran
>
>
> (I'm using 0.8.0-beta1)
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB