Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Transactional writing


+
Tom Brown 2012-10-25, 21:44
+
Neha Narkhede 2012-10-26, 01:19
+
Philip OToole 2012-10-26, 01:31
+
Tom Brown 2012-10-26, 02:04
+
Evan chan 2012-10-26, 02:58
+
Jay Kreps 2012-10-26, 18:08
+
Guozhang Wang 2012-10-26, 18:31
+
Jun Rao 2012-10-29, 03:09
+
Jun Rao 2012-10-29, 05:31
+
Rohit Prasad 2012-11-02, 22:11
+
Tom Brown 2012-11-02, 23:12
+
Rohit Prasad 2012-11-03, 18:51
+
Milind Parikh 2012-11-03, 19:53
+
Jun Rao 2012-10-26, 04:15
+
Tom Brown 2012-10-26, 05:00
+
Jun Rao 2012-10-26, 14:24
+
Tom Brown 2012-10-26, 14:29
+
Jay Kreps 2012-10-26, 18:29
+
Jay Kreps 2012-10-26, 18:18
Copy link to this message
-
Re: Transactional writing
Correct me if I'm wrong, but I thought the intention of kafka was not
really to handle this use case (e.g. transactional writing nor guaranteed
delivery semantics).  Why wouldn't you use a jms queue system (e.g.
activemq) if you need transactional messaging, backed by a database, etc.?

Jason
On Fri, Oct 26, 2012 at 11:18 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> There are a few oddities of using the batching feature to get a kind of
> transactionality (that wasn't the original intention):
> 1. It only actually works if you enable compression. Currently I don't
> think we allow uncompressed recursive message batches. Without this the
> batching only protects against producer failure, in the case of broker
> failure their is no guarantee (either with or without replication) that you
> won't fail in the middle of writing the batch. We could
> consider separating out the batching from the compression support to make
> this work in a more sane way.
> 2. It doesn't work across partitions or topics.
>
> -Jay
>
> On Thu, Oct 25, 2012 at 6:19 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > The closest concept of transaction on the publisher side, that I can
> > think of, is using batch of messages in a single call to the
> > synchronous producer.
> >
> > Precisely, you can configure a Kafka producer to use the "sync" mode
> > and batch messages that require transactional guarantees in a
> > single send() call. That will ensure that either all the messages in
> > the batch are sent or none.
> >
> > Thanks,
> > Neha
> >
> > On Thu, Oct 25, 2012 at 2:44 PM, Tom Brown <[EMAIL PROTECTED]> wrote:
> > > Is there an accepted, or recommended way to make writes to a Kafka
> > > queue idempotent, or within a transaction?
> > >
> > > I can configure my system such that each queue has exactly one
> producer.
> > >
> > > (If there are no accepted/recommended ways, I have a few ideas I would
> > > like to propose. I would also be willing to implement them if needed)
> > >
> > > Thanks in advance!
> > >
> > > --Tom
> >
>
+
Jay Kreps 2012-10-26, 18:47
+
Jonathan Hodges 2013-03-27, 21:42
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB