Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Producer/Consumer questions 0.7


+
Mark 2013-08-29, 18:10
+
Philip OToole 2013-08-29, 20:05
+
Mark 2013-08-29, 20:29
Copy link to this message
-
Re: Producer/Consumer questions 0.7
Philip O'Toole 2013-08-29, 20:41
On Thu, Aug 29, 2013 at 1:28 PM, Mark <[EMAIL PROTECTED]> wrote:

> I'm assuming for this auto-detection of partitions is only with the
> ZookeeperConsumerConnector and not the ConsumerConnector?
>

Plain ol' ConsumerConnector does it for us, in our Java-based high-level
consumers.
>
> On Aug 29, 2013, at 1:04 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Aug 29, 2013 at 11:09 AM, Mark <[EMAIL PROTECTED]>
> wrote:
> >
> >> 1) Should a producer be aware of which broker to write to or is this
> >> somehow managed by Kafka itself. For example, If I have 2 brokers with a
> >> configured partition size of 1 will my messages be written in a
> round-robin
> >> type of fashion to each broker or is this completed dependent on which
> >> broker the consume connects to?
> >>
> >
> > Sounds like a question regarding the Producer, which I am not familiar
> > with. We use our own Producers here, written in C++.
> >
> >
> >>
> >> 2) Similar question goes for a writing to a topic that has multiple
> >> partitions. Are the writes randomly load-balanced between partitions or
> is
> >> this something the client needs to do?
> >>
> >
> > It depends. Producers can specify the partition, or use a partition of
> > '-1', which instructs Kafka to randomly choose a partition.
> >
> >
> >>
> >> 3) If I have 1 topic with 1 partition and 2 brokers how does consuming
> >> messages work. For example, if my consumer connects to broker 1 how
> does it
> >> consume messages written to broker 2?
> >>
> >
> > Don't think of partitions on one broker and partitions on another broker.
> > Simply think of the aggregate partition count across the cluster. The
> > high-level consumer that comes with Kafka will learn where all the
> > partitions are for a given topic via Zookeeper, create connections to the
> > Kafka brokers as needed, and drain all the partitions.
> >
> >
> >>
> >> Thanks
> >>
> >>
>
>

 
+
Mark 2013-08-29, 18:12
+
Philip OToole 2013-08-29, 20:05