Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Setting the consumer's offset


Copy link to this message
-
Re: Setting the consumer's offset
Currently, we have tools like ImportOffset that can update the starting
offset of a consumer in ZK. However, one has to stop all consumers first.

We do plan to allow the high level consumer to specify a starting offset in
the future when we revisit the consumer design. Some of the details are
described in
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design

Thanks,

Jun

On Tue, Apr 2, 2013 at 6:56 AM, James Englert <[EMAIL PROTECTED]> wrote:

> I wrote some code
> here<
> http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+[EMAIL PROTECTED]%3E
> >that
> resets the offset.  You just need to be sure to set this as the
> consumer(s) is starting up.   I'm not certain if it is the "right way" to
> do things.
>
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201303.mbox/%3CCAPD5FitPZkn7Uu+[EMAIL PROTECTED]%3E
>
>
> On Tue, Apr 2, 2013 at 12:46 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Sean,
> >
> > A broker can have multiple topics, each with multiple partitions. Each
> > partition can be consumed by multiple consumers.
> >
> > Our high level consumer API doesn't allow you to specify a starting
> offset.
> > SimpleConsumer does. If you use SimpleConsumer, you are responsible for
> > managing the consumption of all partitions. Consumer group is only used
> in
> > the high level consumer.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson <[EMAIL PROTECTED]> wrote:
> >
> > > Hello,
> > >
> > > Hopefully I'm sending this question to the right place. I'm currently
> > > trying to set up a consumer that will allow me to specify the offset,
> > > partition, and consumer group ID all at the same time. This obviously
> > > causes a dilemma since neither the low-level or high-level consumer
> APIs
> > > seem to support all three. So I have a couple questions:
> > >
> > > 1) Am I correct in my understanding that there is a 1-to-1 relationship
> > > between brokers and partitions? Are multiple consumers per partition
> > > supported? If not, I imagine I could just start up a simple consumer
> per
> > > partition and that would give me what I want. If so, then do simple
> > > consumers actually support consumer group IDs without my knowledge?
> > >
> > > 2) Is it possible to specify a custom offset for the high-level
> consumer
> > > API other than from the tail (autooffset.reset = "smallest") or the
> head
> > > (autooffset.reset = "largest")? I know that the offset for each
> consumer,
> > > topic, and partition relationship is stored in a zookeeper ephemeral
> > node.
> > > Would connecting to zookeeper and changing all these offsets to the
> > desired
> > > values give me what I want? I have tried this with just one consumer
> > using
> > > the high-level api with one partition and one zookeeper host and it
> > seemed
> > > to work alright. But I'm wondering if changing these offsets would work
> > > correctly and efficiently with multiple partitions and consumers with
> the
> > > same ID, as well as multiple zookeeper hosts.
> > >
> > > Thanks,
> > > Sean
> > >
> >
>
>
>
> --
> Jim Englert
> Gilt Groupe
> 2 Park Ave South, 5th Floor
> New York, NY 10011
> M: 847-707-2942
> Please accept my invitation to join Gilt:
> http://www.giltgroupe.com/invite/jenglert
>