Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> message ordering guarantees


+
Ross Black 2013-05-22, 00:42
+
Jun Rao 2013-05-22, 04:19
+
Ross Black 2013-05-22, 04:48
Copy link to this message
-
Re: message ordering guarantees
Yes, reconnect.time.interval.ms is a new config that we added to 0.7. Will
update the website.

VIP is a virtual IP on a load balancer. You can associate multiple brokers
with a VIP and the VIP will randomly select one of the brokers to connect
to during socket establish time. The issue is that if a broker is brought
up, initially there is no connection to it. The reconnect.interval property
gives the producer a chance to pick up the new broker periodically.

Thanks,

Jun
On Tue, May 21, 2013 at 9:48 PM, Ross Black <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> Thanks for the quick response.
> (I just discovered that you had answered a similar question on the dev list
> -
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201304.mbox/%3CCAFbh0Q1XSjG2B9iO%2BpDeM8p9RsDDqKA9E8LY5ScXzAncaQ_mVg%40mail.gmail.com%3E
> )
>
> I set the following properties on the SyncProducer:
>
>         syncProperties.setProperty("reconnect.interval",
> Integer.toString(Integer.MAX_VALUE));
>         syncProperties.setProperty("reconnect.time.interval.ms", "-1");
>
> which resolved the out-of-order problem.
>
> reconnect.time.interval.ms is not documented on the page you specified,
> but
> it was also necessary to stop the periodic reconnect.
>
> We only use a direct connection to the broker using SyncProducer, and don't
> make use of zookeeper.
> Could you please explain what the reconnect is used for (I don't understand
> what VIP is for producers)?
>
> Thanks,
> Ross
>
>
>
> On 22 May 2013 14:19, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > In 0.7, each producer reconnects to the broker periodically. Messages
> sent
> > in different socket connections could arrive out of order. Take a look at
> > reconnect.interval in http://kafka.apache.org/07/configuration.html. If
> > you
> > don't use VIP for the producers, you can actually set reconnect.interval
> to
> > a really large value. In 0.8, there is no reconnect.interval any more.
> > Messages sent from a single producer are expected to arrive in order at
> the
> > broker, as long as they sent to the same partition in a topic.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, May 21, 2013 at 5:42 PM, Ross Black <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi,
> > >
> > > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with
> a
> > > single broker service process.
> > >
> > > I am occasionally seeing messages (from a *single* partition) being
> > > processed out of order to what I expect and I am trying to find where
> the
> > > problem lies.  The problem may well be in my code - I just would like
> to
> > > eliminate Kafka as a potential cause.
> > >
> > > Messages are being sent sequentially from the producer process, using a
> > > single SyncProducer.
> > > Does Kafka provide any guarantees for message ordering in this case?
> > >
> > > eg.  If The sync producer sends messages A then B then C, does the
> Kafka
> > > broker guarantee that messages will be persisted with the order A,B,C?
> > > If not, is there any way to ensure this ordering?
> > > Has anything changed in 0.8 that could be used to ensure that ordering?
> > >
> > >
> > > Thanks,
> > > Ross
> > >
> >
>

 
+
Neha Narkhede 2013-05-22, 21:20
+
Ross Black 2013-05-22, 23:51