Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Lost of messages at C++ Kafka client


Copy link to this message
-
Re: Lost of messages at C++ Kafka client
If I understand what you are asking, I have dealt successfully with the
same type of issue. It can take more than one Boost async_write() over a
broken connection before the client software notices that the connection is
gone.

The best way to detect if a connection is broken is not by detecting that
the ASIO async_write() call returns an error (or raises an exception).
Instead, also start an async_read on the socket (doesn't matter if you
never expect to actually read from it). This call will return immediately
when a connection is broken, allowing much quicker detection of this event.
See if this helps.

Philip

On Wed, Aug 7, 2013 at 2:38 AM, Tianning Zhang <[EMAIL PROTECTED]>wrote:

> Hi,
>
> I am using Kafka (version 0.7.2) for publishing events from some C++
> applications (ca. 20K event /sec). The C++ client is used, which is based
> on asio.
> With the asynchronous and batched messaging, one issue I have is that in
> case Kafka server is broken, the client will not notice the socket lost
> immediately.  Messages go lost between the moment the socket broke and the
> moment when the C++ client issues an error notification.  Message lost is
> not tolerated in our application. Although we have some proprietary
> component to deal with the problem – it is not a convenient solution.
>
> As I understand, this is a known issue. But I would like to know more
> about experiences and best practices for dealing this problems. What can
> future versions of Kafka offer to deal with this problems?
>
> Kind regards
>
> Tianning Zhang ::  [EMAIL PROTECTED]<mailto:
> [EMAIL PROTECTED]>
>
>
>
> [disclaimer logo: ZANOX.de AG]
>
> STRAIGHT TO PERFORMANCE
>
> --------------------------------------------------------------------------------
>
> ZANOX.de AG | Headquarters: Berlin AG Charlottenburg | HRB 75459 | VAT
> identification number: DE 209981705
> Executive Board: Thomas Joosten (CEO) | Stefanie Lüdecke (CSO) | Joachim
> Piroth (CFO) | Christian Rebernik (CTO)
> Chairman of the Supervisory Board: Ralph Büchi
>
> This e-mail and any attachments may contain confidential and/or privileged
> information.
> If you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail from
> your system.
> Any other use, copying, disclosure or distribution is strictly forbidden.
>
> [100 Partnerprogramme 2012]
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB