linkedin and pageview producer + when kafka is down
I'd love to get some insights on how things work at linkedin in terms of
your web servers and kafka producers.
You guys probably connect to multiple kafka clusters, so let's assume you
are only connecting to a single cluster.
1. do you use a single producer for all message types/topics?
2. For your pageview topic i.e. it is getting sent on a per page request
(albiet it is batched):
*What happens when your kafka cluster is down? Will your web application
behave as normal or will it really slow things down?
Locally on my laptop I shutdown my vagrant that is running kafka, and the
page renders very slow when the producer is down.
Or do you use some smart circuit breaker logic that will stop trying to
send producer messages if kafka is down?