Kafka, mail # user - Re: Using Kafka for "data" messages - 2013-06-14, 12:03
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
Re: Using Kafka for "data" messages
Hi Josh,

Thanks for clarifying the use case. The idea is good, but I see the
following three issues

   1. Creating a queue for each user. There could be limits on this
   2. Removing old queues
   3. If the same user logs in from multiple browsers, things get a bit
   more complex.

Can I suggest an alternate approach than using Kafka?

Using a combination of Kafka and XMPP-BOSH/Comet for this.

   1. User logs in. Message is sent on a Kafka queue.
   2. Web browser starts a long polling connection to a server (XMPP-BOSH /
   3. Consumers pick up message in (1) and do their job. They push their
   results to a results queue and to an XMPP end-point ([EMAIL PROTECTED])
   4. Recommender can pick up from the results queue and push it's result
   to the XMPP end-point
   5. Web front-end picks up the messages and does the displaying job.

If you plan it more, you can avoid using Kafka in this use case and just do
with XMPP (for steps 1 and 3)

Also, you don't have to take care of large number of queues, removing them
etc. Also XMPP is really good in handling multiple end-points for a single
user. (There are good XMPP servers like ejabberd and tigase. Also good
lightweight JS libraries for handling connections).

PS: I think my reply is going off-topic. So, I will stop.

On Thu, Jun 13, 2013 at 11:17 PM, Josh Foure <[EMAIL PROTECTED]> wrote:


NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB