Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Is this a good overview of kafka?


Copy link to this message
-
Is this a good overview of kafka?
Just want to verify that I understand the various components correctly,
please chime in where appropriate :)

producer = puts messages on broker(s)
consumer = pulls messages of a broker

In terms of how messages are organized on a cluster of brokers, producers
put messages by providing a "topic".

At the broker side of things, messages are stored by topic but can also be
logicially seperated by a "partition", so that all messages for a
particular topic are directed to a particular broker.

On the consumer side, when you pull messages off, I know you can dedicated
a consumer (or group of consumers) to a particular partition somehow. But
what if you wanted to just randomly pull messages off?  Say I have 3
brokers, and 5 consumers.  How does the consumer know which broker to
connect too, and co-ordinate with the other consumers?

Is there a flow diagram for the above scenerio? (or any other scenerio so I
can understand how the communication takes place).