Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - What are the most significant differences between kestrel and kafka?


Copy link to this message
-
Re: What are the most significant differences between kestrel and kafka?
Evan Chan 2012-10-08, 16:48
Howard,

The Kafka model is very different from the Kestrel model, at least as far
as consumers are concerned.

Kestrel is like AMQP / RabbitMQ in that it keeps track of consumer state,
and you can individually check out messages, and Kestrel will only remove
it from the queue when you "commit".   Keeping track of this state slows
down Kestrel signifiantly though.

Kafka does not keep track of consumer state.  Each consumer keeps track of
an "offset" on its own, and load balancing and offset rollbacks are done
completely in the consumer.   Kafka doesn't support RabbitMQ-style message
checkout and commit, in the server, although you could in theory implement
such a consumer yourself.  The benefit, though is that Kafka has very
little state to track and is wicked fast.

In production, Kafka throughput easily trumps Kestrel throughput when you
use the checkout/commit features.

-Evan
On Mon, Oct 8, 2012 at 9:08 AM, howard chen <[EMAIL PROTECTED]> wrote:

> Hi,
>
> Someone asked in Quora
> (
> http://www.quora.com/Apache-Kafka/What-are-the-most-significant-differences-between-kestrel-and-kafka
> )
> and I found this question is particularly interesting, since not many
> people has the experience in dealing with these systems.
>
> My understanding is as following, so pls correct me if I am wrong:
>
> Same:
> - Both are durable and queue's size is limited by disk storage instead of
> memory
> - There are both very fast
>
> Difference:
>
> - No strict ordering kestrel
> - Not transactional for kestrel
> - No unique delivery guarantee for kestrel
> - No zookeeper coordination for kestrel
>
> So it seems to me that kafka is a more complete solution for message
> streaming solution while kestrel is a simplistic implementation of
> message queue, am I right?
>
> Anything to add?
> Thanks.
>

--
--
*Evan Chan*
Senior Software Engineer |
[EMAIL PROTECTED] | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>