Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Using Kafka for "data" messages

Copy link to this message
Re: Using Kafka for "data" messages
Hi Mahendra, I think that is where it gets a little tricky.  I think it would work something like this:

1.  Web sends login event for user "user123" to topic "GUEST_EVENT".
2.  All of the systems consume those messages and publish the data messages to topic "GUEST_DATA.user123".
3.  The Recommendation system gets all of the data from "GUEST_DATA.user123", processes and then publishes back to the same topic "GUEST_DATA.user123".
4.  The Web consumes the messages from the same topic (there is a different topic for every user that logged in) "GUEST_DATA.user123" and when it finds the recommendation messages it pushes that to the browser (note it will need to read all the other data messages and discard those when looking for the recommendation messages).  I have a concern that the Web will be flooded with a ton of messages that it will promptly drop but I don't want to create a new "response" or "recommendation" topic because then I feel like I am tightly coupling the message to the functionality and in the future different systems may want to consume those messages as well.

Does that make sense?
 From: Mahendra M <[EMAIL PROTECTED]>
Sent: Thursday, June 13, 2013 12:56 PM
Subject: Re: Using Kafka for "data" messages

Hi Josh,

The idea looks very interesting. I just had one doubt.

1. A user logs in. His login id is sent on a topic
2. Other systems (consumers on this topic) consumer this message and
publish their results to another topic

This will be happening without any particular order for hundreds of users.

Now the site being displayed to the user.. How will you fetch only messages
for that user from the queue?


On Thu, Jun 13, 2013 at 8:51 PM, Josh Foure <[EMAIL PROTECTED]> wrote:

> Hi all, my team is proposing a novel
> way of using Kafka and I am hoping someone can help do a sanity check on
> this:
> 1.  When a user logs
> into our website, we will create a “logged in” event message in Kafka
> containing the user id.
> 2.  30+ systems
> (consumers each in their own consumer groups) will consume this event and
> lookup data about this user id.  They
> will then publish all of this data back out into Kafka as a series of data
> messages.  One message may include the user’s name,
> another the user’s address, another the user’s last 10 searches, another
> their
> last 10 orders, etc.  The plan is that a
> single “logged in” event may trigger hundreds if not thousands of
> additional data
> messages.
> 3.  Another system,
> the “Product Recommendation” system, will have consumed the original
> “logged in”
> message and will also consume a subset of the data messages (realistically
> I
> think it would need to consume all of the data messages but would discard
> the
> ones it doesn’t need).  As the Product
> Recommendation consumes the data messages, it will process recommended
> products
> and publish out recommendation messages (that get more and more specific
> as it
> has consumed more and more data messages).
> 4.  The original
> website will consume the recommendation messages and show the
> recommendations to
> the user as it gets them.
> You don’t see many systems implemented this way but since
> Kafka has such a higher throughput than your typical MOM, this approach
> seems
> innovative.
> The benefits are:
> 1.  If we start
> collecting more information about the users, we can simply start publishing
> that in new data messages and consumers can start processing those messages
> whenever they want.  If we were doing
> this in a more traditional SOA approach the schemas would need to change
> every time
> we added a field but with this approach we can just create new messages
> without
> touching existing ones.
> 2.  We are looking to
> make our systems smaller so if we end up with more, smaller systems that
> each
> publish a small number of events, it becomes easier to make changes and