Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> message ordering guarantees


+
Ross Black 2013-05-22, 00:42
+
Jun Rao 2013-05-22, 04:19
+
Ross Black 2013-05-22, 04:48
Copy link to this message
-
Re: message ordering guarantees
Yes, reconnect.time.interval.ms is a new config that we added to 0.7. Will
update the website.

VIP is a virtual IP on a load balancer. You can associate multiple brokers
with a VIP and the VIP will randomly select one of the brokers to connect
to during socket establish time. The issue is that if a broker is brought
up, initially there is no connection to it. The reconnect.interval property
gives the producer a chance to pick up the new broker periodically.

Thanks,

Jun
On Tue, May 21, 2013 at 9:48 PM, Ross Black <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> Thanks for the quick response.
> (I just discovered that you had answered a similar question on the dev list
> -
>
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201304.mbox/%3CCAFbh0Q1XSjG2B9iO%2BpDeM8p9RsDDqKA9E8LY5ScXzAncaQ_mVg%40mail.gmail.com%3E
> )
>
> I set the following properties on the SyncProducer:
>
>         syncProperties.setProperty("reconnect.interval",
> Integer.toString(Integer.MAX_VALUE));
>         syncProperties.setProperty("reconnect.time.interval.ms", "-1");
>
> which resolved the out-of-order problem.
>
> reconnect.time.interval.ms is not documented on the page you specified,
> but
> it was also necessary to stop the periodic reconnect.
>
> We only use a direct connection to the broker using SyncProducer, and don't
> make use of zookeeper.
> Could you please explain what the reconnect is used for (I don't understand
> what VIP is for producers)?
>
> Thanks,
> Ross
>
>
>
> On 22 May 2013 14:19, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > In 0.7, each producer reconnects to the broker periodically. Messages
> sent
> > in different socket connections could arrive out of order. Take a look at
> > reconnect.interval in http://kafka.apache.org/07/configuration.html. If
> > you
> > don't use VIP for the producers, you can actually set reconnect.interval
> to
> > a really large value. In 0.8, there is no reconnect.interval any more.
> > Messages sent from a single producer are expected to arrive in order at
> the
> > broker, as long as they sent to the same partition in a topic.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, May 21, 2013 at 5:42 PM, Ross Black <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi,
> > >
> > > I am using Kafka 0.7.1, and using SyncProducer and SimpleConsumer with
> a
> > > single broker service process.
> > >
> > > I am occasionally seeing messages (from a *single* partition) being
> > > processed out of order to what I expect and I am trying to find where
> the
> > > problem lies.  The problem may well be in my code - I just would like
> to
> > > eliminate Kafka as a potential cause.
> > >
> > > Messages are being sent sequentially from the producer process, using a
> > > single SyncProducer.
> > > Does Kafka provide any guarantees for message ordering in this case?
> > >
> > > eg.  If The sync producer sends messages A then B then C, does the
> Kafka
> > > broker guarantee that messages will be persisted with the order A,B,C?
> > > If not, is there any way to ensure this ordering?
> > > Has anything changed in 0.8 that could be used to ensure that ordering?
> > >
> > >
> > > Thanks,
> > > Ross
> > >
> >
>

 
+
Neha Narkhede 2013-05-22, 21:20
+
Ross Black 2013-05-22, 23:51
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB