On 7 Mar 2014, at 14:11, "Maier, Dr. Andreas" <[EMAIL PROTECTED]> wrote:
You might be interested to have a look at what Samza does: http://samza.incubator.apache.org/learn/documentation/0.7.0/
-- it's a stream processing framework that builds on Kafka's features. It still processes messages sequentially per partition, so it doesn't do the per-message retry that you describe, but it does use a separate Kafka topic for checkpointing state and recovering from failure. (It doesn't require a cascade of topics.)
Indeed. If you want per-message acknowledgement and redelivery, perhaps something like RabbitMQ or ActiveMQ is a better fit for your use case. Kafka's design is optimised for very high-throughput sequential processing of messages, whereas RabbitMQ is better for "job queue" use cases where you want to retry individual messages out-of-order.