Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Using Kafka for "data" messages


Copy link to this message
-
Re: Using Kafka for "data" messages
Hi Mahendra, I think that is where it gets a little tricky.  I think it would work something like this:

1.  Web sends login event for user "user123" to topic "GUEST_EVENT".
2.  All of the systems consume those messages and publish the data messages to topic "GUEST_DATA.user123".
3.  The Recommendation system gets all of the data from "GUEST_DATA.user123", processes and then publishes back to the same topic "GUEST_DATA.user123".
4.  The Web consumes the messages from the same topic (there is a different topic for every user that logged in) "GUEST_DATA.user123" and when it finds the recommendation messages it pushes that to the browser (note it will need to read all the other data messages and discard those when looking for the recommendation messages).  I have a concern that the Web will be flooded with a ton of messages that it will promptly drop but I don't want to create a new "response" or "recommendation" topic because then I feel like I am tightly coupling the message to the functionality and in the future different systems may want to consume those messages as well.

Does that make sense?
Josh
________________________________
 From: Mahendra M <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]; Josh Foure <[EMAIL PROTECTED]>
Sent: Thursday, June 13, 2013 12:56 PM
Subject: Re: Using Kafka for "data" messages
 

Hi Josh,

The idea looks very interesting. I just had one doubt.

1. A user logs in. His login id is sent on a topic
2. Other systems (consumers on this topic) consumer this message and
publish their results to another topic

This will be happening without any particular order for hundreds of users.

Now the site being displayed to the user.. How will you fetch only messages
for that user from the queue?

Regards,
Mahendra

On Thu, Jun 13, 2013 at 8:51 PM, Josh Foure <[EMAIL PROTECTED]> wrote:

Mahendra

http://twitter.com/mahendra
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB