Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> RE: message order, guarenteed?


+
S Ahmed 2013-06-10, 00:33
+
Neha Narkhede 2013-06-10, 12:37
+
David Arthur 2013-06-14, 14:37
Copy link to this message
-
Re: message order, guarenteed?
Another idea. If a set of messages arrive over a single TCP connection, route to a partition depending on TCP connection.

To be honest, these approaches, while they work, may not scale when the message rate is high. If at all possible, try to think of a way to remove this requirement from your design. For example, a design might have a sequence number assigned to each message before it goes into Kafka (a time-based UUID, for example), and something later in the pipe line sorts it all out. Kafka then does what is does best, IMHO, a high-performance reliable, message bus.

Philip

On Jun 14, 2013, at 7:37 AM, David Arthur <[EMAIL PROTECTED]> wrote:

> Simple example of how to take advantage of this behavior:
>
> Suppose you're sending document updates through Kafka. If you use the document ID as the message key and the default hash partitioner, the updates for a given document will exist on the same partition and come into the consumer in order.
>
> On 6/10/13 8:37 AM, Neha Narkhede wrote:
>> Kafka guarantees order per topic  partition per source client.
>>
>> Thanks,
>> Neha
>> On Jun 9, 2013 5:33 PM, "S Ahmed" <[EMAIL PROTECTED]> wrote:
>>
>>> I understand that there are no guarantees per say that a message may be a
>>> duplicate (its the consumer's job to guarantee that), but when it comes to
>>> message order, is kafka built in such a way that it is impossible to get
>>> messages in the wrong order?
>>>
>>> Certain use cases might not be sensitive to order, but when order is very
>>> important, is kafka the wrong tool for the job or is there a way to get
>>> this requirement?
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB