Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Transactional writing


+
Tom Brown 2012-10-25, 21:44
+
Neha Narkhede 2012-10-26, 01:19
Copy link to this message
-
Re: Transactional writing
On Thu, Oct 25, 2012 at 06:19:04PM -0700, Neha Narkhede wrote:
> The closest concept of transaction on the publisher side, that I can
> think of, is using batch of messages in a single call to the
> synchronous producer.
>
> Precisely, you can configure a Kafka producer to use the "sync" mode
> and batch messages that require transactional guarantees in a
> single send() call. That will ensure that either all the messages in
> the batch are sent or none.

This is an interesting feature -- something I wasn't aware of. Still it
doesn't solve the problem *completely*. As many people realise, it's still
possible for the batch of messages to get into Kafka fine, but the ack from
Kafka to be lost on its way back to the Producer. In that case the Producer
erroneously believes the messages didn't get in, and might re-send them.

You guys *haven't* solved that issue, right? I believe you write about it on
the Kafka site.

>
> Thanks,
> Neha
>
> On Thu, Oct 25, 2012 at 2:44 PM, Tom Brown <[EMAIL PROTECTED]> wrote:
> > Is there an accepted, or recommended way to make writes to a Kafka
> > queue idempotent, or within a transaction?
> >
> > I can configure my system such that each queue has exactly one producer.
> >
> > (If there are no accepted/recommended ways, I have a few ideas I would
> > like to propose. I would also be willing to implement them if needed)
> >
> > Thanks in advance!
> >
> > --Tom

--
Philip O'Toole

Senior Developer
Loggly, Inc.
San Francisco, Calif.
www.loggly.com

Come join us!
http://loggly.com/company/careers/
+
Tom Brown 2012-10-26, 02:04
+
Evan chan 2012-10-26, 02:58
+
Jay Kreps 2012-10-26, 18:08
+
Guozhang Wang 2012-10-26, 18:31
+
Jun Rao 2012-10-29, 03:09
+
Jun Rao 2012-10-29, 05:31
+
Rohit Prasad 2012-11-02, 22:11
+
Tom Brown 2012-11-02, 23:12
+
Rohit Prasad 2012-11-03, 18:51
+
Milind Parikh 2012-11-03, 19:53
+
Jun Rao 2012-10-26, 04:15
+
Tom Brown 2012-10-26, 05:00
+
Jun Rao 2012-10-26, 14:24
+
Tom Brown 2012-10-26, 14:29
+
Jay Kreps 2012-10-26, 18:29
+
Jay Kreps 2012-10-26, 18:18
+
Jason Rosenberg 2012-10-26, 18:29
+
Jay Kreps 2012-10-26, 18:47
+
Jonathan Hodges 2013-03-27, 21:42