If your required.acks is 1 then the producer only ensures that one broker receives the data before it's sucessfully returned to the client.
Therefore if the broker crashes and lost all the data then you lose data, or similarly it can happen even before the data is fsynced.
To ensure there are more copies of your data in case of failure scenarios you want to increase your required.acks to more than 1 to tolerate failuries.
Also async producer doesn't wait until the data is sent before it returns, as it buffers and writes asynchronously. To ensure each write that has a succesful response is written you want to use the sync producer.
On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <[EMAIL PROTECTED]> wrote:
2014-06-03 21:46:09 INFO Producer:68 - Shutting down producer 2014-06-03 21:46:09 INFO ProducerSendThread:68 - Begin shutting down ProducerSendThread 2014-06-03 21:46:09 INFO ProducerSendThread:68 - Shutdown ProducerSendThread complete 2014-06-03 21:46:09 INFO ProducerPool:68 - Closing all sync producers On Jun 3, 2014, at 9:58 PM, Timothy Chen <[EMAIL PROTECTED]> wrote:
We are converting JSON to Avro and that conversion is done by the custom serializer.
Our volume calculation on the producer side is based on the AVRO generic record that is passed to the producer send method— not of the encoded data output from the serializer that is what actual got send the Broker I believe.
That could be the gap and I am testing now without the customer serializer and seeing the two volumes are very close. That could be it!!
On Jun 3, 2014, at 7:22 PM, Jun Rao <[EMAIL PROTECTED]> wrote: