Subject: Is this a good overview of kafka?
Just want to verify that I understand the various components correctly,
please chime in where appropriate :)
producer = puts messages on broker(s)
consumer = pulls messages of a broker
In terms of how messages are organized on a cluster of brokers, producers
put messages by providing a "topic".
At the broker side of things, messages are stored by topic but can also be
logicially seperated by a "partition", so that all messages for a
particular topic are directed to a particular broker.
On the consumer side, when you pull messages off, I know you can dedicated
a consumer (or group of consumers) to a particular partition somehow. But
what if you wanted to just randomly pull messages off? Say I have 3
brokers, and 5 consumers. How does the consumer know which broker to
connect too, and co-ordinate with the other consumers?
Is there a flow diagram for the above scenerio? (or any other scenerio so I
can understand how the communication takes place).