My organization has a redundant setup across 2 data-centers. During normal operations both data-centers are up and handling workloads. For whatever reason, the infrastucture team has decided to operate 2 independent Kubernetes clusters, one in each DC.
We’ve just started developing a cloud-native application that runs as a sharded cluster and is deployed on Kubernetes. This application reads from an event stream and writes to a DB and more event streams.
Our plan is to deploy two application clusters (one per DC), and keep them running in parallel, in active-active mode, where both clusters are actively reading everything from the event stream and writing to the DB simultaneously. They both do all the work, and the first one to write it’s output wins.
While this costs more hardware resources because everything is performed twice (once per DC), it also has some important advantages (zero downtime on DC failure, smoothing out transient latency spikes from one DC, no split brain issues across DCs).
So here are the questions:
Does this kind of architecture with competing application instances have a name?
We plan to write to a Postgre DB. Is there a way to perform this kind of highly concurrent updates where we expect everthing to be written twice from different transactions, without getting too many errors/rollbacks? Would MERGE statements, or ON CONFLICT clauses work?
6
To answer the Postgres part of the question, assuming that you are entering records with a well defined primary key, using something like ON CONFLICT DO NOTHING would work well, as it would simply ignore the conflicting record.
Edit: To clarify, by well defined I really mean something that will be the same no matter which cluster it is inserted from.
Just be more careful if the record uses some kind of randomly generated key which is created by the service or something, which is somewhat common in high throughput event streams, as obviously this will not work the same way and you may need to implement more logic to handle the conflict and this will make the process slower.