I have a kubernetes cluster which needs to be replaced (not upgraded), and I’m trying to figure out how to do it as seamlessly as possible. In other situations I just destroy the old cluster and launch the new one, but in this case, there will be workloads running in the old cluster that need to be allowed to run to completion.

When I try to search for tips on this, I usually get information about upgrading a cluster, or upgrading apps/services within the cluster with minimal downtime. Not much about cluster replacement.

The plan I have is:

1. create a new cluster, “cluster-2”.
2. All variables that reference the old cluster need to reference “cluster-2” so that new workloads start there.
3. When nothing is running in the old cluster, destroy it.

Is there a better way to do this?

background:

• cluster is running in GKE.
• cluster replacement is necessary as we are required to change some config options that GKE says are immutable, so the cluster cannot be modified in place, it needs to be recreated. This might need to happen again in the future.
• we use pulumi for managing these resources, but I don’t think pulumi can easily handle this situation, though if there are tips to make this easier with pulumi I’d be open to them.