Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Setting the consumer's offset


Copy link to this message
-
Re: Setting the consumer's offset
Jun Rao 2013-04-02, 04:47
Sean,

A broker can have multiple topics, each with multiple partitions. Each
partition can be consumed by multiple consumers.

Our high level consumer API doesn't allow you to specify a starting offset.
SimpleConsumer does. If you use SimpleConsumer, you are responsible for
managing the consumption of all partitions. Consumer group is only used in
the high level consumer.

Thanks,

Jun

On Mon, Apr 1, 2013 at 7:10 PM, Sean Grayson <[EMAIL PROTECTED]> wrote:

> Hello,
>
> Hopefully I'm sending this question to the right place. I'm currently
> trying to set up a consumer that will allow me to specify the offset,
> partition, and consumer group ID all at the same time. This obviously
> causes a dilemma since neither the low-level or high-level consumer APIs
> seem to support all three. So I have a couple questions:
>
> 1) Am I correct in my understanding that there is a 1-to-1 relationship
> between brokers and partitions? Are multiple consumers per partition
> supported? If not, I imagine I could just start up a simple consumer per
> partition and that would give me what I want. If so, then do simple
> consumers actually support consumer group IDs without my knowledge?
>
> 2) Is it possible to specify a custom offset for the high-level consumer
> API other than from the tail (autooffset.reset = "smallest") or the head
> (autooffset.reset = "largest")? I know that the offset for each consumer,
> topic, and partition relationship is stored in a zookeeper ephemeral node.
> Would connecting to zookeeper and changing all these offsets to the desired
> values give me what I want? I have tried this with just one consumer using
> the high-level api with one partition and one zookeeper host and it seemed
> to work alright. But I'm wondering if changing these offsets would work
> correctly and efficiently with multiple partitions and consumers with the
> same ID, as well as multiple zookeeper hosts.
>
> Thanks,
> Sean
>