Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Question about resetting offsets and the high-level consumer


+
Bob Cotton 2012-11-09, 13:04
+
Jun Rao 2012-11-09, 15:30
+
Michal Haris 2012-11-10, 11:40
+
Michal Haris 2012-11-10, 14:41
+
Jun Rao 2012-11-12, 06:03
+
Michal Haris 2012-11-14, 00:08
+
Jun Rao 2012-11-14, 05:38
Copy link to this message
-
Re: Question about resetting offsets and the high-level consumer
Well, resetting could follow the partition ownership where only owned
partitions can be reset by any one consumer. There's definitely a case for
consumer applications that are more about observing frequencies rather than
total aggregation and so are typically interested in last couple of seconds
/ minutes of the stream.

On 14 November 2012 05:38, Jun Rao <[EMAIL PROTECTED]> wrote:

> It's more than that. For example, what if 2 consumers want to set different
> offsets for the same partition?
>
> Jun
>
> On Tue, Nov 13, 2012 at 4:08 PM, Michal Haris <[EMAIL PROTECTED]
> >wrote:
>
> > Of course assuming that none of the consumers in the group are interested
> > in the missed messages in such case.
> > On Nov 12, 2012 6:03 AM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> >
> > > It's easy if there is only a single consumer in the group. In general,
> > > however, there could be multiple consumers in the same group. So, some
> > kind
> > > of coordination is needed.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Nov 10, 2012 at 6:41 AM, Michal Haris <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Found it in archives, autooffset.reset is only used when the consumer
> > > > registering first time or the last committed offset is out-of-range,
> so
> > > > that means that we have to create direct connection to the zookeeper
> > and
> > > > set/delete the /consumer/xyz/offset/topic123/*-* before starting the
> > > > consumer to get it reset to the head of the stream.. shouldn't be a
> big
> > > > problem to patch the consumer api with a method to do that right ?
> > > >
> > > > On 10 November 2012 11:40, Michal Haris <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > Really? So what does the autooffset.reset=largest do ?
> > > > > On Nov 9, 2012 3:30 PM, "Jun Rao" <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > >> Bob,
> > > > >>
> > > > >> The high level consumer doesn't support resetting the starting
> > offset
> > > > >> right
> > > > >> now. We will look into that post 0.8 as part of the consumer
> > redesign.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >>
> > > > >> On Fri, Nov 9, 2012 at 5:04 AM, Bob Cotton <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >>
> > > > >> > We have a topic whose consumers, on startup, only need to
> consume
> > > from
> > > > >> the
> > > > >> > head of the topic. They don't need to consume old messages they
> > may
> > > > have
> > > > >> > missed.
> > > > >> >
> > > > >> > We would like to use the high-level consumer, as the API is a
> bit
> > > > >> simpler.
> > > > >> >
> > > > >> > Is there a way to reset the current offsets for the high-level
> > > > consumer?
> > > > >> >
> > > > >> > We've though about deleting the consumer group nodes in ZK
> > manually,
> > > > but
> > > > >> > that seems brittle as those locations may change from release to
> > > > >> release.
> > > > >> >
> > > > >> > Is there this level of control somewhere I've missed?
> > > > >> >
> > > > >> > Thanks
> > > > >> > - Bob
> > > > >> >
> > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > Michal Haris
> > > > Software Engineer
> > > >
> > > > www.visualdna.com | t: +44 (0) 207 734 7033
> > > >
> > >
> >
>

--
Michal Haris
Software Engineer

www.visualdna.com | t: +44 (0) 207 734 7033
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB