Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> kafka producer - retry messages


+
Demian Berjman 2013-11-28, 14:38
+
Otis Gospodnetic 2013-11-28, 23:58
+
Philip OToole 2013-11-29, 00:09
+
Steve Morin 2013-11-29, 00:12
+
Philip OToole 2013-11-29, 01:06
+
Philip OToole 2013-11-29, 00:15
Copy link to this message
-
Re: kafka producer - retry messages
Philip,
  How would do you mirror this to a main Kafka instance?
-Steve

> On Nov 28, 2013, at 16:14, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
> I should add in our custom producers we buffer in RAM if required, so Kafka can be restarted etc. But I would never code streaming to disk now. I would just run a Kafka instance on the same node.
>
> Philip
>
>> On Nov 28, 2013, at 4:08 PM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>>
>> By FS I guess you mean file system.
>>
>> In that case, if one is that concerned, why not run a single Kafka broker on the same machine, and connect to it over localhost? And disable ZK mode too, perhaps.
>>
>> I may be missing something, but I never fully understand why people try really hard to build a stream-to-disk backup approach, when they might be able to couple tightly to Kafka, which, well, just streams to disk.
>>
>> Philip
>>
>>> On Nov 28, 2013, at 3:58 PM, Otis Gospodnetic <[EMAIL PROTECTED]> wrote:
>>>
>>> Hi,
>>>
>>> We've done this at Sematext, where we use Kafka in all 3 products/services
>>> you see in my signature.  When we fail to push a message into Kafka we
>>> store it in the FS and from there we can process it later.
>>>
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>>
>>> On Thu, Nov 28, 2013 at 9:37 AM, Demian Berjman <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi.
>>>>
>>>> Anyone has build a retry system (durable) for the producer in case the
>>>> kakfa cluster is down? We have certain messages that must be sent because
>>>> there are after a transaction that cannot be undo.
>>>>
>>>> We could set the property "message.send.max.retries", but if the producer
>>>> goes down, we lost the messages.
>>>>
>>>> Thanks,
>>>>

 
+
Philip OToole 2013-11-29, 00:56
+
Diego Parra 2013-11-29, 00:40
+
Philip OToole 2013-11-29, 01:04
+
Otis Gospodnetic 2013-11-29, 01:57
+
Joe Stein 2013-11-28, 15:11
+
Demian Berjman 2013-11-28, 15:28
+
Steve Morin 2013-11-28, 15:38
+
Demian Berjman 2013-11-28, 16:07
+
Steve Morin 2013-11-28, 16:14
+
Demian Berjman 2013-11-28, 16:36
+
Steve Morin 2013-11-28, 22:52