-
Re: Error Handling and Acknowledgements in Kafka ConsumersFelix GV 2012-10-15, 23:26
One thing you could do is the following:
Every time you consume a message, you send it asynchronously to a pool of
actor that will process the message.
When the actors fail to process a message (or if it takes them longer than
a certain arbitrary time-out period), they republish it to another topic
and move on. You can then have another consumer that takes care of those
problematic messages, while the main consumer keeps going with the easy to
process messages.
This would allow you to not be blocked by messages that take a long time to
process.
However, this system is not as reliable, because the Kafka consumer will
assume that a message is correctly consumed (and move the offset
accordingly) as soon as it is sent to the actor pool, even though the
message is not necessarily processed yet. In certain failure scenarios,
some messages would end up never being consumed at all.
Another limitation is that such system doesn't allow you to ensure that the
actors process the messages in the same order Kafka consumes them.
I guess it could work for certain use cases...
--
Felix
On Thu, Sep 13, 2012 at 5:10 PM, Vito Sabella <[EMAIL PROTECTED]> wrote:
> ** Resending due to outlook.com corrupting the original message. Sorry :)
> **
>
> Kafka-users,
>
> I'm looking to use kafka in a pub-sub model where a consumer reads
> from Kafka and does some processing on the message. How would you
> recommend a commit to Zookeeper / setting the last message consumed
> location if processing one of the messages in the pipe is more
> unreliable than the others.
>
> Let's say I read a batch of 10 messages (1-10) and I successfully
> process messages 1-8 and 10 quickly, but message #9 is taking an
> inordinately long time to process. I don't want to write the message
> as consumed against Zookeeper but I also don't want to block forward
> progress of the pipeline for that topic/partition.
>
> In the edge case, let's say processing all messages was successful but
> processing message #9 totally failed/timed out, but a reattempt at
> processing it would not result in failure (web-service call, strange
> network condition). In the timeout/failure case would the suggestion
> be to re-queue the message?
>
> Does anyone have any recommendations for the above scenarios?
>
> Thanks
> Vito
>
Every time you consume a message, you send it asynchronously to a pool of
actor that will process the message.
When the actors fail to process a message (or if it takes them longer than
a certain arbitrary time-out period), they republish it to another topic
and move on. You can then have another consumer that takes care of those
problematic messages, while the main consumer keeps going with the easy to
process messages.
This would allow you to not be blocked by messages that take a long time to
process.
However, this system is not as reliable, because the Kafka consumer will
assume that a message is correctly consumed (and move the offset
accordingly) as soon as it is sent to the actor pool, even though the
message is not necessarily processed yet. In certain failure scenarios,
some messages would end up never being consumed at all.
Another limitation is that such system doesn't allow you to ensure that the
actors process the messages in the same order Kafka consumes them.
I guess it could work for certain use cases...
--
Felix
On Thu, Sep 13, 2012 at 5:10 PM, Vito Sabella <[EMAIL PROTECTED]> wrote:
> ** Resending due to outlook.com corrupting the original message. Sorry :)
> **
>
> Kafka-users,
>
> I'm looking to use kafka in a pub-sub model where a consumer reads
> from Kafka and does some processing on the message. How would you
> recommend a commit to Zookeeper / setting the last message consumed
> location if processing one of the messages in the pipe is more
> unreliable than the others.
>
> Let's say I read a batch of 10 messages (1-10) and I successfully
> process messages 1-8 and 10 quickly, but message #9 is taking an
> inordinately long time to process. I don't want to write the message
> as consumed against Zookeeper but I also don't want to block forward
> progress of the pipeline for that topic/partition.
>
> In the edge case, let's say processing all messages was successful but
> processing message #9 totally failed/timed out, but a reattempt at
> processing it would not result in failure (web-service call, strange
> network condition). In the timeout/failure case would the suggestion
> be to re-queue the message?
>
> Does anyone have any recommendations for the above scenarios?
>
> Thanks
> Vito
>
