I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with a single broker service process.
I am occasionally seeing messages (from a *single* partition) being processed out of order to what I expect and I am trying to find where the problem lies. The problem may well be in my code - I just would like to eliminate Kafka as a potential cause.
Messages are being sent sequentially from the producer process, using a single SyncProducer. Does Kafka provide any guarantees for message ordering in this case?
eg. If The sync producer sends messages A then B then C, does the Kafka broker guarantee that messages will be persisted with the order A,B,C? If not, is there any way to ensure this ordering? Has anything changed in 0.8 that could be used to ensure that ordering? Thanks, Ross
In 0.7, each producer reconnects to the broker periodically. Messages sent in different socket connections could arrive out of order. Take a look at reconnect.interval in http://kafka.apache.org/07/configuration.html. If you don't use VIP for the producers, you can actually set reconnect.interval to a really large value. In 0.8, there is no reconnect.interval any more. Messages sent from a single producer are expected to arrive in order at the broker, as long as they sent to the same partition in a topic.
Jun On Tue, May 21, 2013 at 5:42 PM, Ross Black <[EMAIL PROTECTED]> wrote:
reconnect.time.interval.ms is not documented on the page you specified, but it was also necessary to stop the periodic reconnect.
We only use a direct connection to the broker using SyncProducer, and don't make use of zookeeper. Could you please explain what the reconnect is used for (I don't understand what VIP is for producers)?
On 22 May 2013 14:19, Jun Rao <[EMAIL PROTECTED]> wrote:
Yes, reconnect.time.interval.ms is a new config that we added to 0.7. Will update the website.
VIP is a virtual IP on a load balancer. You can associate multiple brokers with a VIP and the VIP will randomly select one of the brokers to connect to during socket establish time. The issue is that if a broker is brought up, initially there is no connection to it. The reconnect.interval property gives the producer a chance to pick up the new broker periodically.
Jun On Tue, May 21, 2013 at 9:48 PM, Ross Black <[EMAIL PROTECTED]> wrote: