Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> understanding partitions based on wiki example of profile visits


Copy link to this message
-
Re: understanding partitions based on wiki example of profile visits
Yes, your description is correct. A particular member's data would all be
in one partition.

Broker partitions are just the unit of parallelism--think of each partition
as a totally ordered log you can append to and read from. The consumption
of one of these partition logs is single threaded.

The guarantee is that all messages are added to a partition in the order
they arrive. From the point of view of a single producer client this will
also be the order in which they are sent. These messages are then delivered
in this order to a consumer thread.

Hope that helps.

-Jay
On Sun, Nov 25, 2012 at 7:54 PM, S Ahmed <[EMAIL PROTECTED]> wrote:

> The wiki states "Consider an application that would like to maintain an
> aggregation of the number of profile visitors for each member. It would
> like to send all profile visit events for a member to a particular
> partition and, hence, have all updates for a member to appear in the same
> stream for the same consumer thread." (
> http://incubator.apache.org/kafka/design.html)
>
> So say I have 5 broker servers, now my producer will send a message for a
> particular profile page visit, with the default algorithm using
> hash(member_id)%num_partitions
> to figur out which broker server to send it it.
>
> So a particular members pageview messages will all go to a single server
> then, is this the case?  And therefore all the messages for a given user
> will be in the correct order also right?
>
> So a consumer group that subscribes to the 'profile-page-view' topic will
> consume page view related messages, is it possible to subscribe to a
> particular broker partition also?
>
> Are broker partitions meant for cases when you want all messages to be
> saved on the same node?
>