I need to write an application that consumes data from Kafka. The data is produced with the default partition scheme. The application cares about the current message and most recent message for any given key. In the event of the application going down (scaling in, deployment, failure), the most recent message would need to be fetched somehow.

I’m considering using a database for this – read and write the DB every time a message is received. This seems like it might be complex, potentially prone to errors and slow.

I’m also wondering if this can be achieved by configuring the consumer to read from the earliest offset available. I think there would be a bit of a performance hit for every rebalance that happens but it may be overcome with log compaction.