Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> RE: message order, guarenteed?

Copy link to this message
Re: message order, guarenteed?
Another idea. If a set of messages arrive over a single TCP connection, route to a partition depending on TCP connection.

To be honest, these approaches, while they work, may not scale when the message rate is high. If at all possible, try to think of a way to remove this requirement from your design. For example, a design might have a sequence number assigned to each message before it goes into Kafka (a time-based UUID, for example), and something later in the pipe line sorts it all out. Kafka then does what is does best, IMHO, a high-performance reliable, message bus.


On Jun 14, 2013, at 7:37 AM, David Arthur <[EMAIL PROTECTED]> wrote:

> Simple example of how to take advantage of this behavior:
> Suppose you're sending document updates through Kafka. If you use the document ID as the message key and the default hash partitioner, the updates for a given document will exist on the same partition and come into the consumer in order.
> On 6/10/13 8:37 AM, Neha Narkhede wrote:
>> Kafka guarantees order per topic  partition per source client.
>> Thanks,
>> Neha
>> On Jun 9, 2013 5:33 PM, "S Ahmed" <[EMAIL PROTECTED]> wrote:
>>> I understand that there are no guarantees per say that a message may be a
>>> duplicate (its the consumer's job to guarantee that), but when it comes to
>>> message order, is kafka built in such a way that it is impossible to get
>>> messages in the wrong order?
>>> Certain use cases might not be sensitive to order, but when order is very
>>> important, is kafka the wrong tool for the job or is there a way to get
>>> this requirement?