I wish to use either the ConsumerConnector or the SimpleConsumer to
read messages from all partitions across multiple brokers using a
fixed number of threads (hopefully leveraging asynchronous IO for high
I know that the ConsumerConnector sounds like this, but the
documentation was not clear about a few things. Would somebody who has
used it (or knows the code) be willing to answer these questions about
- Does the ConsumerConnector manage connections to multiple brokers,
or just a single broker?
- Does the ConsumerConnector require a thread for each partition on
each broker? (If not, how many threads does it require?)
- Does the ConsumerConnector use actual asynchronous IO, or does it
mimic it by using a dedicated behind-the-scenes thread (and the
traditional java socket API)?
If the ConsumerConnector won't serve my purpose I think I could use
the SimpleConsumer to implement a version of this, but I worry that
it's not thread safe; That is, can I use the same SimpleConsumer
instance on Thread-A, then on Thread-B, and then again on Thread-A
(though never at the same time)?
Thanks in advance!