Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Commit Offset per topic


Copy link to this message
-
Re: Commit Offset per topic
Joel. Just to be clear consumer re-balances you mentioning in the FAQ refers to
consumer restart, am I correct?
On Wed, Nov 6, 2013 at 5:51 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:

> This question seems to come up often - added this to the FAQ.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon%27twantmyconsumer%27soffsetstobecommittedautomatically.CanImanuallymanagemyconsumer%27soffsets%3F
>
> On Tue, Nov 05, 2013 at 04:02:09PM +0000, Roman Garcia wrote:
> > Thanks Neha! I guess auto-commit it is for now...
> >
> >
> > On Tue, Nov 5, 2013 at 5:08 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> > > Currently, the only way to achieve that is to use the SimpleConsumer
> API.
> > > We are considering the feature you mentioned for the 0.9 release -
> > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Mon, Nov 4, 2013 at 9:57 AM, Roman Garcia <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Right Neha, thanks. I was aware of that particular API, but I process
> > > > topics within separate threads (actors) so I cannot assume every
> > > > topic/partition is supposed to commit its offset. Instead, I could
> use a
> > > > per-stream (topic-partition) commit.
> > > > Is this use case valid? Should there be a ticket to implement this?
> > > > Regards,
> > > > Roman
> > > >
> > > >
> > > > On Mon, Nov 4, 2013 at 1:28 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > The API you are looking for is ConsumerConnector.commitOffsets().
> > > > However,
> > > > > not that it will commit offsets for all partitions that the
> particular
> > > > > consumer connector owns at that time. And the set of partitions
> owned
> > > by
> > > > a
> > > > > particular consumer connector can change over time.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Mon, Nov 4, 2013 at 7:58 AM, Roman Garcia <
> [EMAIL PROTECTED]>
> > > > > wrote:
> > > > >
> > > > > > Hi, I'm still using Kafka 0.7, and was wondering if there's a
> way I
> > > can
> > > > > > "manually" commit offsets per Topic/Partition.
> > > > > >
> > > > > > I was expecting something like KafkaMessageStream#commitOffset()
> > > > > > I couldn't find an API for this. If there isn't any, is there a
> > > reason?
> > > > > >
> > > > > > Regards,
> > > > > > Roman
> > > > > >
> > > > >
> > > >
> > >
>
>