Cut costs on GKE

[georgethebeatle]

Background

There is a community initiative to cut on our GKE costs. Here is the related issue in the community repo: cloudfoundry/community#1189

Action to take

• Reduce the number of nightly periodic test runs
• Move test clusters to a cheaper zone
• Move acceptance to a cheaper zone
• Move concourse to a cheaper zone
• Delete unused resource
• Address additional suggestions from community issue
• Downsize cluster nodes if possible. Cluster config is no longer shared, so individual sizing is possible.

Impact

Cut the GKE costs for the cf on k8s working group

Dev Notes

• Be careful with vault and the secrets stored in it
• Be careful with the concourse logs
• For the concourse cluster we should first create the new one, install vault and concourse and migrate the secrets and logs to the new place before destroying the old one.
• We should be careful with the urls of concourse and vault - maybe we should move the old ones to a different url, so that the original DNS names are free to be allocated to the new ones
• We could delete postfacto and its sql database to save a bit more, as we are generally not using it. Postfacto is a legacy app anyway and could break at any time.

[georgethebeatle]

The cheapest zone right now is europe-north2. It has a couple of regions, all of which seem to be offering the e2 instances that we are using, so I guess it is okay to move to europe-north2-a.

Our CI pipeline has a single switch for the zone, but might be a good idea to also change the defaults in our terraform scripts as we can't be sure that it is not hitting the default in some cases. When changing the region and zone values we have to be very careful. We should first destroy with the old value, then create with the new value. This might mean multiple back and forth commits in both korifi-ci and the terraform configs.

[georgethebeatle]

Everything is migrated to europe-north2. The old concourse cluster and the old database are not yet destroyed, but the old database is stopped to double check that everything is working without it. If all is well we can destroy the redundant resources after a few days and it will become clear how much we are saving.

Additionally the nightly periodic test runs have been halved in number which should save cpu

Here is what is left to be done

• Destroy old concourse cluster
• Destroy old concourse-db
• Merge the europe-north2 branch in the secret repo to main, so that we converge back to a single state of the world
• Set the build cluster to concourse-north

[georgethebeatle]

Downsized the concourse database from 4core/26GB to 2core/13GB. The PointFive tooling recommendations were to go to 3core/16GB and to reduce storage to 36GB. However it turns out that these recommendations cannot be implementes because of limitations in GKE:

• Storage cannot shrink, but grow only. If we really want to shrink it we must create a new database and manyally migrate the data. This poses risk of data loss that is not justified by the promised saving of $10 per month.
• Core number should be an even number, so 3 is not an option. Memory seems to be tied to the number of CPUs, so we can chose betweed 4cpu/26GB and 2cpu/13GB. We went for the latter for the time being to test it out. The PointFive suggestion is larger than 2cpu/13GB, so we might have to go back to the bigger vm type.

[georgethebeatle]

After migration to europe-north2 we are observing ~10% decrease in costs.

Another possible optimization that we can try is to run periodic tests until midnight, then destroy test cluster and create it in the morning, this way the two test clusters (gke and eks) will not be around 24/7 and will potentially cost less. We were hoping that reducing the nuber of nightly test runs will drop the price, but this is not the case, so this new approach might help.