Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Proper use of ConsumerConnector


Copy link to this message
-
Proper use of ConsumerConnector
I wish to use either the ConsumerConnector or the SimpleConsumer to
read messages from all partitions across multiple brokers using a
fixed number of threads (hopefully leveraging asynchronous IO for high
performance).

I know that the ConsumerConnector sounds like this, but the
documentation was not clear about a few things. Would somebody who has
used it (or knows the code) be willing to answer these questions about
it:

- Does the ConsumerConnector manage connections to multiple brokers,
or just a single broker?
- Does the ConsumerConnector require a thread for each partition on
each broker? (If not, how many threads does it require?)
- Does the ConsumerConnector use actual asynchronous IO, or does it
mimic it by using a dedicated behind-the-scenes thread (and the
traditional java socket API)?

If the ConsumerConnector won't serve my purpose I think I could use
the SimpleConsumer to implement a version of this, but I worry that
it's not thread safe; That is, can I use the same SimpleConsumer
instance on Thread-A, then on Thread-B, and then again on Thread-A
(though never at the same time)?
Thanks in advance!

--Tom