I can think of two options of implementing below requirement and request
some guidance on choosing the option with pros and cons.
- A in memory rules cache to be loaded from log compacted kafka topic. This
cache has to be loaded prior to arrival of events.
- Updates to the log compacted kafka topic has to be tracked to keep the in
memory rule cache up to date
Additional properties of data:
- On Job start/restart, this rule cache is always loaded from earliest
available offset in the log. - No kafka offset store and restore required.
- No checkpointing needed for the rule cache, as it is loaded afresh in
event of crash and restore
- No eventTime semantics required as we always want the latest rules to be
loaded to cache
1. Using a KafkaConsumer in open() doing a initial load, and continuously
fetching rule updates and keeping the in memory cache up to date. This
option is not using a DataStream for rules as we don't use any goodies of
stream like state,checkpoint, event time etc.
2. Connected Stream approach. Using a KafkaConsumer in open() doing a
initial load. Have a FlinkKafkaSource Stream connected with events. In this
case have to take care of out of order updates to caches, since the rules
updates are from open() and Rule DataStream.