Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Proper use of ConsumerConnector


Copy link to this message
-
Proper use of ConsumerConnector
I wish to use either the ConsumerConnector or the SimpleConsumer to
read messages from all partitions across multiple brokers using a
fixed number of threads (hopefully leveraging asynchronous IO for high
performance).

I know that the ConsumerConnector sounds like this, but the
documentation was not clear about a few things. Would somebody who has
used it (or knows the code) be willing to answer these questions about
it:

- Does the ConsumerConnector manage connections to multiple brokers,
or just a single broker?
- Does the ConsumerConnector require a thread for each partition on
each broker? (If not, how many threads does it require?)
- Does the ConsumerConnector use actual asynchronous IO, or does it
mimic it by using a dedicated behind-the-scenes thread (and the
traditional java socket API)?

If the ConsumerConnector won't serve my purpose I think I could use
the SimpleConsumer to implement a version of this, but I worry that
it's not thread safe; That is, can I use the same SimpleConsumer
instance on Thread-A, then on Thread-B, and then again on Thread-A
(though never at the same time)?
Thanks in advance!

--Tom

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB