Home | About | Sematext search-lucene.com search-hadoop.com search-devops.com metrics + logs = try SPM and Logsene for free
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> log file flushing, flushall on clean shutdown...


Copy link to this message
-
Re: log file flushing, flushall on clean shutdown...
buffered socket channel, that received the bytes, but didn't actually send
them out at the os level.  No exception is thrown, so I am wondering
whether this socket is ever flushed, or if it just quietly fails to send
its data.  Thoughts?

That's correct. When the producer sends some data, it enters the os
socket buffer on the producer. Once it is flushed, it is then buffered
in the server's socket buffer before being read by the server and
appended to the log. On producer or server shutdown, messages can be
lost since they are not flushed from the kernel's socket buffer. In
0.7, you will not notice any error. In 0.8, your producer will wait to
read the response from the server and either timeout or throw a broken
pipe exception. In this case, you can choose to retry.

I think you are referring to yet another issue that only happens on
the 0.7 zookeeper based producer. If you shutdown any broker, messages
in the async producer's queue are lost. This is because, it load
balances and picks a target broker when the message enters the queue.
So even if it can detect the broker is gone, it cannot change the
target broker for the messages in the producer's queue.

Both of these issues are fixed in 0.8.

Thanks,
Neha

On Thu, Mar 28, 2013 at 11:44 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB