Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> understanding partitions based on wiki example of profile visits


Copy link to this message
-
understanding partitions based on wiki example of profile visits
The wiki states "Consider an application that would like to maintain an
aggregation of the number of profile visitors for each member. It would
like to send all profile visit events for a member to a particular
partition and, hence, have all updates for a member to appear in the same
stream for the same consumer thread." (
http://incubator.apache.org/kafka/design.html)

So say I have 5 broker servers, now my producer will send a message for a
particular profile page visit, with the default algorithm using
hash(member_id)%num_partitions
to figur out which broker server to send it it.

So a particular members pageview messages will all go to a single server
then, is this the case?  And therefore all the messages for a given user
will be in the correct order also right?

So a consumer group that subscribes to the 'profile-page-view' topic will
consume page view related messages, is it possible to subscribe to a
particular broker partition also?

Are broker partitions meant for cases when you want all messages to be
saved on the same node?
+
S Ahmed 2012-11-26, 14:45
+
Jay Kreps 2012-11-26, 16:53
+
S Ahmed 2012-11-26, 17:19
+
Jay Kreps 2012-11-26, 19:02
+
S Ahmed 2012-11-27, 20:00
+
Jay Kreps 2012-11-27, 21:35