Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Proper use of ConsumerConnector


+
Tom Brown 2012-12-19, 18:49
Copy link to this message
-
Re: Proper use of ConsumerConnector
In general, you should use the consumer connector - unless you have a good
reason to load balance and manage offsets manually (which is taken care of
in the consumer connector).
- Does the ConsumerConnector manage connections to multiple brokers,
> or just a single broker?
>

Multiple brokers.
> - Does the ConsumerConnector require a thread for each partition on
> each broker? (If not, how many threads does it require?)
>

You can specify how many streams you want - if there are more partitions
than threads, then a given thread can consume from multiple partitions. If
there are more threads than available partitions, there will be idle
threads.
> - Does the ConsumerConnector use actual asynchronous IO, or does it
> mimic it by using a dedicated behind-the-scenes thread (and the
> traditional java socket API)?
>

The consumer connector uses SimpleConsumers for each broker that it
connects to. These consumers fetch from each broker and insert chunks into
blocking queues which the consumer iterators then dequeue.

Joel

 
+
永辉 赵 2012-12-20, 09:07
+
Neha Narkhede 2012-12-20, 16:56
+
Joel Koshy 2012-12-20, 18:06
+
Neha Narkhede 2012-12-20, 18:14
+
Tom Brown 2012-12-20, 19:32
+
Neha Narkhede 2012-12-20, 19:43
+
永辉 赵 2012-12-21, 10:56
+
Neha Narkhede 2012-12-21, 17:37
+
Tom Brown 2012-12-21, 17:47
+
Jun Rao 2012-12-21, 17:57
+
Yonghui Zhao 2012-12-21, 23:12
+
Tom Brown 2012-12-22, 01:44
+
Neha Narkhede 2012-12-23, 19:54