Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Is this a good overview of kafka?


Copy link to this message
-
Is this a good overview of kafka?
Just want to verify that I understand the various components correctly,
please chime in where appropriate :)

producer = puts messages on broker(s)
consumer = pulls messages of a broker

In terms of how messages are organized on a cluster of brokers, producers
put messages by providing a "topic".

At the broker side of things, messages are stored by topic but can also be
logicially seperated by a "partition", so that all messages for a
particular topic are directed to a particular broker.

On the consumer side, when you pull messages off, I know you can dedicated
a consumer (or group of consumers) to a particular partition somehow. But
what if you wanted to just randomly pull messages off?  Say I have 3
brokers, and 5 consumers.  How does the consumer know which broker to
connect too, and co-ordinate with the other consumers?

Is there a flow diagram for the above scenerio? (or any other scenerio so I
can understand how the communication takes place).

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB