Skip to content

Update install instructions from guided-setup #490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HappyTetrahedron merged 1 commit into from
Jul 31, 2026
Merged

Update install instructions from guided-setup #490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 92 additions & 39 deletions docs/modules/ROOT/partials/guided-setup/cloudscale-decommission.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,21 @@
* &nbsp;&nbsp;<<step-11,And I disable machine autoscaling>>
* &nbsp;&nbsp;<<step-12,And I delete all persistent volumes>>
* &nbsp;&nbsp;<<step-13,And I delete all machinesets>>
* <<step-14,Then I save the loadbalancer metadata>>
* &nbsp;&nbsp;<<step-15,And I downtime the loadbalancers in icinga>>
* &nbsp;&nbsp;<<step-16,And I decommission Terraform resources>>
* <<step-17,Then I delete Cloudscale server groups>>
* &nbsp;&nbsp;<<step-18,And I delete all S3 buckets>>
* &nbsp;&nbsp;<<step-19,And I delete the cluster backup>>
* &nbsp;&nbsp;<<step-20,And I delete the cluster's API tokens>>
* &nbsp;&nbsp;<<step-21,And I remove the LoadBalancers from control.vshn.net>>
* &nbsp;&nbsp;<<step-22,And I decommission the LoadBalancers>>
* &nbsp;&nbsp;<<step-23,And I remove the cluster's DNS entries>>
* <<step-24,Then I delete the cluster's Vault secrets>>
* &nbsp;&nbsp;<<step-25,And I delete the cluster's OpsGenie heartbeat>>
* &nbsp;&nbsp;<<step-26,And I delete the cluster from Lieutenant>>
* &nbsp;&nbsp;<<step-27,And I delete the Keycloak service>>
* <<step-14,Then I compile the cluster catalog>>
* <<step-15,Then I save the loadbalancer metadata>>
* &nbsp;&nbsp;<<step-16,And I downtime the loadbalancers in icinga>>
* &nbsp;&nbsp;<<step-17,And I decommission Terraform resources>>
* <<step-18,Then I delete Cloudscale server groups>>
* &nbsp;&nbsp;<<step-19,And I delete all S3 buckets>>
* &nbsp;&nbsp;<<step-20,And I delete the cluster backup>>
* &nbsp;&nbsp;<<step-21,And I delete the cluster's API tokens>>
* &nbsp;&nbsp;<<step-22,And I remove the LoadBalancers from control.vshn.net>>
* &nbsp;&nbsp;<<step-23,And I decommission the LoadBalancers>>
* &nbsp;&nbsp;<<step-24,And I remove the cluster's DNS entries>>
* <<step-25,Then I delete the cluster's Vault secrets>>
* &nbsp;&nbsp;<<step-26,And I delete the cluster's OpsGenie heartbeat>>
* &nbsp;&nbsp;<<step-27,And I delete the cluster from Lieutenant>>
* &nbsp;&nbsp;<<step-28,And I delete the Keycloak service>>

[[step-1]]
=== Given I have all prerequisites installed
Expand Down Expand Up @@ -609,6 +610,34 @@ kubectl get nodes
----

[[step-14]]
=== Then I compile the cluster catalog

This step compiles the catalog to ensure you have a local checkout of the cluster repository
during decommissioning.

==== Inputs

* `commodore_cluster_id`

==== Script

[source,bash]
----
OUTPUT=$(mktemp)

# export INPUT_commodore_cluster_id=

set -euo pipefail
commodore catalog compile ${INPUT_commodore_cluster_id}


# echo "# Outputs"
# cat "$OUTPUT"
# rm -f "$OUTPUT"

----

[[step-15]]
=== Then I save the loadbalancer metadata

This step gathers metadata on the LoadBalancer instances (such as their icinga zone and backup server), such that they can be properly decommissioned down the line.
Expand Down Expand Up @@ -730,7 +759,7 @@ popd

----

[[step-15]]
[[step-16]]
=== And I downtime the loadbalancers in icinga

In this step you have to configure downtimes in Icinga for the cluster's load balancers.
Expand Down Expand Up @@ -769,7 +798,7 @@ sleep 2

----

[[step-16]]
[[step-17]]
=== And I decommission Terraform resources

This step decommissions all Terraform resources for the cluster.
Expand Down Expand Up @@ -861,7 +890,7 @@ popd

----

[[step-17]]
[[step-18]]
=== Then I delete Cloudscale server groups

This step cleans up server groups configured for this cluster on Cloudscale.
Expand Down Expand Up @@ -897,7 +926,7 @@ done

----

[[step-18]]
[[step-19]]
=== And I delete all S3 buckets

This step deletes the cluster's associated S3 buckets from Cloudscale.
Expand Down Expand Up @@ -946,7 +975,7 @@ mc rb "${INPUT_commodore_cluster_id}/${INPUT_commodore_cluster_id}-logstore" --f

----

[[step-19]]
[[step-20]]
=== And I delete the cluster backup

This step deletes the cluster's associated backup bucket from Cloudscale.
Expand Down Expand Up @@ -1069,7 +1098,7 @@ curl -i -H "Authorization: Bearer ${INPUT_cloudscale_token}" -X DELETE "$(echo "

----

[[step-20]]
[[step-21]]
=== And I delete the cluster's API tokens

This step deletes the cluster's associated Cloudscale API tokens from Cloudscale.
Expand Down Expand Up @@ -1119,7 +1148,7 @@ echo " Success."

----

[[step-21]]
[[step-22]]
=== And I remove the LoadBalancers from control.vshn.net

In this step you need to remove the LoadBalancer servers from control.vshn.net
Expand All @@ -1128,7 +1157,13 @@ In this step you need to remove the LoadBalancer servers from control.vshn.net

* `lb_fqdn_1`
* `lb_fqdn_2`
* `control_vshn_api_token`
* `control_vshn_api_token`: Your personal token for the control.vshn.net Servers API.
Be extra careful with the IP allowlist.

If you can't be bothered to find/make one, put in a bogus value.
The spell will then guide you to manually delete the definitions.



==== Script

Expand All @@ -1141,9 +1176,27 @@ OUTPUT=$(mktemp)
# export INPUT_control_vshn_api_token=

set -euo pipefail
echo "Deleting server definitions ..."
curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/appuio/${INPUT_lb_fqdn_1}
curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/appuio/${INPUT_lb_fqdn_2}

AUTH="X-AccessToken: ${INPUT_control_vshn_api_token}"

code="$( curl -H"$AUTH" https://control.vshn.net/api/servers/1/appuio/ -o /dev/null -w"%{http_code}" )"
if [[ "$code" != 200 ]]
then
echo '###################################################################################'
echo '# #'
echo "# Please manually delete the cluster's LoadBalancer servers before proceeding. #"
echo '# #'
echo '###################################################################################'
echo
echo You can go to:
echo https://control.vshn.net/servers/definitions/appuio/${INPUT_lb_fqdn_1}/delete
echo https://control.vshn.net/servers/definitions/appuio/${INPUT_lb_fqdn_2}/delete
sleep 2
exit 0
fi

curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/_/"${INPUT_lb_fqdn_1}"
curl -XDELETE -H"$AUTH" https://control.vshn.net/api/servers/1/_/"${INPUT_lb_fqdn_2}"
echo "Server definitions deleted."


Expand All @@ -1153,7 +1206,7 @@ echo "Server definitions deleted."

----

[[step-22]]
[[step-23]]
=== And I decommission the LoadBalancers

This step decommissions resources associated with the Puppet managed LoadBalancers.
Expand Down Expand Up @@ -1185,9 +1238,9 @@ OUTPUT=$(mktemp)
set -euo pipefail
echo "# Clearing encdata caches ... #"
# shellcheck disable=2029
ssh nfs1.ch1.puppet.vshn.net "sudo rm /srv/nfs/export/puppetserver-puppetserver-enc-cache-pvc-*/${INPUT_lb_fqdn_1}.yaml" || true
ssh master8.puppet.vshn.net "sudo rm /etc/puppetlabs/code/enc/${INPUT_lb_fqdn_1}.yaml" || true
# shellcheck disable=2029
ssh nfs1.ch1.puppet.vshn.net "sudo rm /srv/nfs/export/puppetserver-puppetserver-enc-cache-pvc-*/${INPUT_lb_fqdn_2}.yaml" || true
ssh master8.puppet.vshn.net "sudo rm /etc/puppetlabs/code/enc/${INPUT_lb_fqdn_2}.yaml" || true
echo "# Cleared encdata caches. #"
echo
echo "# Cleaning up LBs in icinga ... #"
Expand Down Expand Up @@ -1259,13 +1312,13 @@ for lb in "${INPUT_lb_fqdn_1}" "${INPUT_lb_fqdn_2}"
do
for backup_server in "${INPUT_lb_backup_1}" "${INPUT_lb_backup_2}"
do
# shellcheck disable=2029
ssh "$backup_server" "sudo rm /var/lib/burp/CA/${lb}.crt" || true
# shellcheck disable=2029
ssh "$backup_server" "sudo rm /var/lib/burp/CA/${lb}.csr" || true
# shellcheck disable=2029,2015
ssh "$backup_server" "sudo rm /var/lib/burp/CA/${lb}.crt" && echo "Deleted ${lb}.crt from ${backup_server}." || true
# shellcheck disable=2029,2015
ssh "$backup_server" "sudo rm /var/lib/burp/CA/${lb}.csr" && echo "Deleted ${lb}.csr from ${backup_server}." || true
backup="/var/lib/burp/${lb}"
# shellcheck disable=2029
ssh "$backup_server" "sudo rm -rf ${backup}" || true
# shellcheck disable=2029,2015
ssh "$backup_server" "sudo rm -rf ${backup}" && echo "Deleted backup for ${lb} (if it existed) from ${backup_server}." || true
done
done
echo "# Deleted backups from Burp server. #"
Expand All @@ -1277,7 +1330,7 @@ echo "# Deleted backups from Burp server. #"

----

[[step-23]]
[[step-24]]
=== And I remove the cluster's DNS entries

In this step, you must manually remove any DNS entries associated with the cluster from https://git.vshn.net/vshn/vshn_zonefiles.
Expand All @@ -1304,7 +1357,7 @@ sleep 2

----

[[step-24]]
[[step-25]]
=== Then I delete the cluster's Vault secrets

This step cleans up all the cluster's Vault secrets.
Expand Down Expand Up @@ -1376,7 +1429,7 @@ done

----

[[step-25]]
[[step-26]]
=== And I delete the cluster's OpsGenie heartbeat

This step deletes the cluster's OpsGenie heartbeat.
Expand Down Expand Up @@ -1415,7 +1468,7 @@ curl "https://api.opsgenie.com/v2/heartbeats/${INPUT_commodore_cluster_id}" \

----

[[step-26]]
[[step-27]]
=== And I delete the cluster from Lieutenant

This step deletes the cluster from Lieutenant
Expand Down Expand Up @@ -1447,7 +1500,7 @@ curl -H "Authorization: Bearer $(commodore fetch-token)" -XDELETE \

----

[[step-27]]
[[step-28]]
=== And I delete the Keycloak service

This step deletes the cluster's keycloak service from control.vshn.net
Expand Down
Loading