eg: Our event contains a timestamp so it is ok to have data out of order into partition due to parallel write to Brokers per partition... We are looking to dump the messages parallel not build up queue while it data is being send to brokers and the send thread will take time to complete entire batch if we have round robin on Partitions ( eg message count % number Of partitions).
Please let me know if there is any alternative.. Thanks, Bhavesh
The concerns you expressed are fixed in the new producer implementation available on trunk. You can give it a try at org.apache.kafka.clients.producer. On Tue, Jun 24, 2014 at 4:37 PM, Bhavesh Mistry <[EMAIL PROTECTED]> wrote:
I am referring to version 0.8.0 and I would like to try in dev environment unbounded queue to try to figure upper bound of the application during the stress test and through put we want to achieve.
Also, additional features I would like to add At Run-time on Producer side :
1) Ability to change the tuning parameters at Run-time (eg Batch Size, Queue size, enqueue time etc) because our app run on various private cloud , physical box (Same app) so it would be great If Kafka allow tuning parameter to be reset at runtime.
2) During the batch Async send, occasionally we get MessageSizeTooLargeException, is it possible ensure from the producer side that batch can not exceed configure limit, and if it does exceed the limit, divide the batch into 1/2 and send it transparently unless single message exceed the limit.
Our use case is "transferring log lines", we can configure batch size for optimal throughput but we do not want to loose data when batch size exceed configured limit (2MB). It there any way we can introduce this batch split strategy logic.
3) Purpose of injecting the Queue implementation, we want to preserve a latest message in queue and remove the oldest ( Priority Queue or drop message oldest message). Because by configuring the enqueue timeout, it drops latest message.
eg. we have some monitoring message that take greater priority than regular "log Lines" etc...
I hope this clarify the use cases.
On Wed, Jun 25, 2014 at 12:41 PM, Steve Morin <[EMAIL PROTECTED]> wrote: