Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - consuming only half the messages produced


Copy link to this message
-
Re: consuming only half the messages produced
Jun Rao 2013-05-02, 05:15
Partition is different from replicas. A topic can have one or more
partitions and each partition can have one or more replicas. A consumer
consumes data at partition level. In other words, a consumer gets the same
data no matter how many replicas are there.

When you say the consumer only gets half of the messages, do you mean that
it gets half of the messages that are produced?

You may want to take a look at the consumer example in
http://kafka.apache.org/08/api.html

Thanks,

Jun
On Wed, May 1, 2013 at 7:14 PM, Rob Withers <[EMAIL PROTECTED]> wrote:

> Running a consumer group (createStreams()), pointing to the zookeeper and
> with the topic and 1 consumer thread, results in only half the messages
> being consumed.  The topic was auto-created, with a replication factor of
> 2,
> but the producer was configured to produce to 2 brokers and so 4 partitions
> resulted.  Are half getting sent to one leader, in one broker, and the
> other
> half getting sent to another leader, in the other broker, but the consumer
> stream is only reading from one leader from the zk?  Shouldn't there only
> be
> one leader?
>
>
>
> thanks,
>
> rob
>
>
>
>