Release-candidate: uninstall.sh: namespace stuck in `Terminating` due to un-finalized Grafana CRs

[amd-ama10002-2]

Summary

Running SaFE/bootstrap/uninstall.sh leaves the primus-safe namespace stuck in Terminating indefinitely. The blocking resources are leftover Grafana custom resources (GrafanaDashboard, GrafanaDatasource) that still carry the operator.grafana.com/finalizer. Once the Grafana operator is uninstalled, nothing remains to run that finalizer, so namespace deletion never completes.

Target branch: this fix should land on the release-candidate branch experimental/v1-ga-separation.

Environment

• Script: SaFE/bootstrap/uninstall.sh
• Run with PURGE_PVC=true (full clean-slate path, which deletes namespaces)

Steps to reproduce

1. Install SaFE with the Grafana operator / Grafana CRs present.
2. Run bash uninstall.sh and answer y to the PVC purge prompt (or PURGE_PVC=true ./uninstall.sh).
3. Observe kubectl get ns primus-safe remains Terminating.

Evidence

kubectl get ns primus-safe -o jsonpath='{.status.conditions}':

• NamespaceContentRemaining = True: Some resources are remaining: grafanadashboards.grafana.integreatly.org has 1 resource instances, grafanadatasources.grafana.integreatly.org has 1 resource instances
• NamespaceFinalizersRemaining = True: Some content in the namespace has finalizers remaining: operator.grafana.com/finalizer in 2 resource instances

Root cause

Teardown ordering / coverage gap. The script uninstalls the Grafana operator:

helm uninstall grafana-operator -n primus-safe 2>/dev/null || true

(SaFE/bootstrap/uninstall.sh line 70) but never (a) deletes the Grafana CRs while the operator is still running to finalize them, nor (b) force-clears their finalizers as a safety net. Once the operator pod is gone, the operator.grafana.com/finalizer on the remaining GrafanaDashboard / GrafanaDatasource CRs can never be run, wedging the namespace in Terminating.

This is the same class of bug the script already handles for the other operators. For VictoriaMetrics / OpenSearch / FluentBit it correctly:

• deletes the CRs FIRST (while operators are alive to garbage-collect), and
• includes a finalizer safety net that patches finalizers: [] on any residual CRs (see the VM_CR_KINDS loop and the opensearchcluster/fluentbit patch loop, lines ~34-68).

There is no equivalent handling for the Grafana CRs.

Scope confirmed against the repo

• Namespace: The Grafana CRs are created by the primus-safe chart into .Release.Namespace, i.e. primus-safe (not primus-safe-observability). See SaFE/charts/primus-safe/templates/grafana/*.yaml.
• Kinds actually deployed by SaFE: Grafana, GrafanaDashboard, GrafanaDatasource.
• Full CR set the grafana-operator manages (SaFE/charts/grafana-operator/crds/), for a robust delete + finalizer-clear loop:
  grafanas, grafanadashboards, grafanadatasources, grafanafolders, grafanalibrarypanels, grafanacontactpoints, grafanaalertrulegroups, grafanamutetimings, grafananotificationpolicies, grafananotificationpolicyroutes, grafananotificationtemplates, grafanaserviceaccounts.

Immediate workaround (unstick a currently-hung namespace)

for kind in grafanadashboards grafanadatasources; do
  for cr in $(kubectl -n primus-safe get "$kind" -o name 2>/dev/null); do
    kubectl -n primus-safe patch "$cr" --type=merge \
      -p '{"metadata":{"finalizers":[]}}'
  done
done

Proposed fix

Mirror the existing operator-teardown pattern for Grafana in uninstall.sh:

1. Before helm uninstall grafana-operator, delete the Grafana CRs in primus-safe so the operator can finalize them, e.g.:
   kubectl -n primus-safe delete grafanadashboard,grafanadatasource,grafana --all --ignore-not-found --timeout=120s
   (extend to the full CR kind list above for completeness).
2. Add a finalizer safety net (patch finalizers: []) on any residual Grafana CRs, matching the opensearchcluster/fluentbit/VM loops, in case the operator was already gone.

Acceptance criteria

• Running uninstall.sh (with PURGE_PVC=true) leaves no namespace stuck in Terminating when Grafana CRs are present.
• Re-running uninstall.sh is still idempotent / safe when the CRs or operator are already gone.