Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Producer/Consumer questions 0.7


+
Mark 2013-08-29, 18:10
+
Philip OToole 2013-08-29, 20:05
+
Mark 2013-08-29, 20:29
Copy link to this message
-
Re: Producer/Consumer questions 0.7
On Thu, Aug 29, 2013 at 1:28 PM, Mark <[EMAIL PROTECTED]> wrote:

> I'm assuming for this auto-detection of partitions is only with the
> ZookeeperConsumerConnector and not the ConsumerConnector?
>

Plain ol' ConsumerConnector does it for us, in our Java-based high-level
consumers.
>
> On Aug 29, 2013, at 1:04 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
> > On Thu, Aug 29, 2013 at 11:09 AM, Mark <[EMAIL PROTECTED]>
> wrote:
> >
> >> 1) Should a producer be aware of which broker to write to or is this
> >> somehow managed by Kafka itself. For example, If I have 2 brokers with a
> >> configured partition size of 1 will my messages be written in a
> round-robin
> >> type of fashion to each broker or is this completed dependent on which
> >> broker the consume connects to?
> >>
> >
> > Sounds like a question regarding the Producer, which I am not familiar
> > with. We use our own Producers here, written in C++.
> >
> >
> >>
> >> 2) Similar question goes for a writing to a topic that has multiple
> >> partitions. Are the writes randomly load-balanced between partitions or
> is
> >> this something the client needs to do?
> >>
> >
> > It depends. Producers can specify the partition, or use a partition of
> > '-1', which instructs Kafka to randomly choose a partition.
> >
> >
> >>
> >> 3) If I have 1 topic with 1 partition and 2 brokers how does consuming
> >> messages work. For example, if my consumer connects to broker 1 how
> does it
> >> consume messages written to broker 2?
> >>
> >
> > Don't think of partitions on one broker and partitions on another broker.
> > Simply think of the aggregate partition count across the cluster. The
> > high-level consumer that comes with Kafka will learn where all the
> > partitions are for a given topic via Zookeeper, create connections to the
> > Kafka brokers as needed, and drain all the partitions.
> >
> >
> >>
> >> Thanks
> >>
> >>
>
>

 
+
Mark 2013-08-29, 18:12
+
Philip OToole 2013-08-29, 20:05
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB