Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Lost of messages at C++ Kafka client


Copy link to this message
-
Re: Lost of messages at C++ Kafka client
If I understand what you are asking, I have dealt successfully with the
same type of issue. It can take more than one Boost async_write() over a
broken connection before the client software notices that the connection is
gone.

The best way to detect if a connection is broken is not by detecting that
the ASIO async_write() call returns an error (or raises an exception).
Instead, also start an async_read on the socket (doesn't matter if you
never expect to actually read from it). This call will return immediately
when a connection is broken, allowing much quicker detection of this event.
See if this helps.

Philip

On Wed, Aug 7, 2013 at 2:38 AM, Tianning Zhang <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am using Kafka (version 0.7.2) for publishing events from some C++
> applications (ca. 20K event /sec). The C++ client is used, which is based
> on asio.
> With the asynchronous and batched messaging, one issue I have is that in
> case Kafka server is broken, the client will not notice the socket lost
> immediately.  Messages go lost between the moment the socket broke and the
> moment when the C++ client issues an error notification.  Message lost is
> not tolerated in our application. Although we have some proprietary
> component to deal with the problem – it is not a convenient solution.
>
> As I understand, this is a known issue. But I would like to know more
> about experiences and best practices for dealing this problems. What can
> future versions of Kafka offer to deal with this problems?
>
> Kind regards
>
> Tianning Zhang ::  [EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>
>
>
>
> [disclaimer logo: ZANOX.de AG]
>
> STRAIGHT TO PERFORMANCE
>
> --------------------------------------------------------------------------------
>
> ZANOX.de AG | Headquarters: Berlin AG Charlottenburg | HRB 75459 | VAT
> identification number: DE 209981705
> Executive Board: Thomas Joosten (CEO) | Stefanie Lüdecke (CSO) | Joachim
> Piroth (CFO) | Christian Rebernik (CTO)
> Chairman of the Supervisory Board: Ralph Büchi
>
> This e-mail and any attachments may contain confidential and/or privileged
> information.
> If you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail from
> your system.
> Any other use, copying, disclosure or distribution is strictly forbidden.
>
> [100 Partnerprogramme 2012]
>