Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Transactional writing


+
Tom Brown 2012-10-25, 21:44
+
Neha Narkhede 2012-10-26, 01:19
+
Philip OToole 2012-10-26, 01:31
+
Tom Brown 2012-10-26, 02:04
+
Evan chan 2012-10-26, 02:58
+
Jay Kreps 2012-10-26, 18:08
+
Guozhang Wang 2012-10-26, 18:31
+
Jun Rao 2012-10-29, 03:09
+
Jun Rao 2012-10-29, 05:31
+
Rohit Prasad 2012-11-02, 22:11
+
Tom Brown 2012-11-02, 23:12
+
Rohit Prasad 2012-11-03, 18:51
+
Milind Parikh 2012-11-03, 19:53
+
Jun Rao 2012-10-26, 04:15
+
Tom Brown 2012-10-26, 05:00
+
Jun Rao 2012-10-26, 14:24
+
Tom Brown 2012-10-26, 14:29
+
Jay Kreps 2012-10-26, 18:29
+
Jay Kreps 2012-10-26, 18:18
Copy link to this message
-
Re: Transactional writing
Correct me if I'm wrong, but I thought the intention of kafka was not
really to handle this use case (e.g. transactional writing nor guaranteed
delivery semantics).  Why wouldn't you use a jms queue system (e.g.
activemq) if you need transactional messaging, backed by a database, etc.?

Jason
On Fri, Oct 26, 2012 at 11:18 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> There are a few oddities of using the batching feature to get a kind of
> transactionality (that wasn't the original intention):
> 1. It only actually works if you enable compression. Currently I don't
> think we allow uncompressed recursive message batches. Without this the
> batching only protects against producer failure, in the case of broker
> failure their is no guarantee (either with or without replication) that you
> won't fail in the middle of writing the batch. We could
> consider separating out the batching from the compression support to make
> this work in a more sane way.
> 2. It doesn't work across partitions or topics.
>
> -Jay
>
> On Thu, Oct 25, 2012 at 6:19 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > The closest concept of transaction on the publisher side, that I can
> > think of, is using batch of messages in a single call to the
> > synchronous producer.
> >
> > Precisely, you can configure a Kafka producer to use the "sync" mode
> > and batch messages that require transactional guarantees in a
> > single send() call. That will ensure that either all the messages in
> > the batch are sent or none.
> >
> > Thanks,
> > Neha
> >
> > On Thu, Oct 25, 2012 at 2:44 PM, Tom Brown <[EMAIL PROTECTED]> wrote:
> > > Is there an accepted, or recommended way to make writes to a Kafka
> > > queue idempotent, or within a transaction?
> > >
> > > I can configure my system such that each queue has exactly one
> producer.
> > >
> > > (If there are no accepted/recommended ways, I have a few ideas I would
> > > like to propose. I would also be willing to implement them if needed)
> > >
> > > Thanks in advance!
> > >
> > > --Tom
> >
>
+
Jay Kreps 2012-10-26, 18:47
+
Jonathan Hodges 2013-03-27, 21:42