Kafka, mail # user - Is this a good overview of kafka? - 2013-01-14, 19:16
 Search Hadoop and all its subprojects:
Copy link to this message
-
Is this a good overview of kafka?
Just want to verify that I understand the various components correctly,
please chime in where appropriate :)

producer = puts messages on broker(s)
consumer = pulls messages of a broker

In terms of how messages are organized on a cluster of brokers, producers
put messages by providing a "topic".

At the broker side of things, messages are stored by topic but can also be
logicially seperated by a "partition", so that all messages for a
particular topic are directed to a particular broker.

On the consumer side, when you pull messages off, I know you can dedicated
a consumer (or group of consumers) to a particular partition somehow. But
what if you wanted to just randomly pull messages off?  Say I have 3
brokers, and 5 consumers.  How does the consumer know which broker to
connect too, and co-ordinate with the other consumers?

Is there a flow diagram for the above scenerio? (or any other scenerio so I
can understand how the communication takes place).

 
+
Felix GV 2013-01-14, 23:09
+
Stan Rosenberg 2013-01-15, 02:38
+
Felix GV 2013-01-15, 05:25
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB