Connect() runs before the WasDeleted check, so a rejected ProviderConfigUsage create can block deletion of a managed resource in a terminating namespace

[mattlykeaya]

What happened?

On a Crossplane v2 cluster (v2.3.3, upjet-based Azure providers v2.7.0) we observed a namespace take hours to finish terminating. During that time:

• Azure providers issued 825× HTTP 403 attempting to create providerconfigusages.azure.m.upbound.io in the terminating namespace, with reason: NamespaceTerminating ("unable to create new content in namespace ... because it is being terminated"). RBAC allowed every one of these — the rejection is namespace lifecycle, not permissions.
• The kube namespace-controller swept the namespace's content (deletecollection, all resource types) at a sustained ~31.6/s the entire time.

Both stopped together the moment the namespace finally finished terminating.

Root cause (traced in crossplane-runtime)

1. In pkg/reconciler/managed/reconciler.go, Reconcile() calls r.external.Connect(externalCtx, managed) unconditionally — before the meta.WasDeleted(managed) branch that leads to external.Delete() and RemoveFinalizer(). This is true even when deletionPolicy: Delete and the managed resource already has a deletion timestamp.
2. Upjet-generated providers' Connect() implementations call ProviderConfigUsageTracker.Track() (pkg/resource/providerconfig.go) first, which does a Get-then-Create-or-Update Apply of the resource's ProviderConfigUsage.
3. Track()'s only swallowed error class is IsNotAllowed (pkg/resource/resource.go) — an internal sentinel produced by its own AllowUpdateIf predicate, not apierrors.IsForbidden. A genuine 403 from the API server is not swallowed.
4. If the ProviderConfigUsage doesn't currently exist — e.g. it was already removed by the namespace controller's deletecollection sweep of that GVR — and the namespace is Terminating, the Create leg gets a 403 (NamespaceTerminating). That error propagates out of Track() → out of Connect().
5. The reconciler's Connect() error handling (reconciler.go, right after the Connect call) returns early with Requeue: true — before ever reaching the deletion branch. external.Delete() and RemoveFinalizer() never run that cycle.
6. Because the managed resource's own finalizer survives, the namespace never fully drains, so the namespace controller re-sweeps every resource type again, which can strip a freshly-recreated ProviderConfigUsage before the next reconcile — restarting the race.

The loop only breaks when a Track() create happens to land in a timing gap between namespace-controller sweeps, so resolution looks like a race, not a guarantee. We could not establish an upper bound on how long this can take, or whether it can wedge indefinitely under sustained contention.

How to reproduce

1. Create a namespace with a namespaced (Crossplane v2) managed resource that references a ProviderConfig/ClusterProviderConfig, using any upjet-based provider.
2. Let it reconcile normally so its ProviderConfigUsage exists.
3. Delete the ProviderConfigUsage directly (simulating the namespace-controller sweep having already removed it) at roughly the same time as kubectl delete namespace <ns> (or delete the namespace first, then delete the PCU before the next reconcile).
4. Observe: the managed resource's Connect() fails with a NamespaceTerminating 403 while attempting to recreate the ProviderConfigUsage; the managed resource's finalizer is not removed while this keeps failing; the namespace stays Terminating far longer than expected.

Expected behavior

Deleting a namespace holding Crossplane managed resources should complete in bounded, predictable time. At minimum:

• Track()/Connect() should not need to succeed in order for a managed resource already being deleted to proceed to external.Delete() and finalizer removal — or the reconciler should check WasDeleted() before calling Connect() when deletion doesn't require re-establishing a fresh ProviderConfigUsage.
• Alternatively, a NamespaceTerminating 403 encountered while tracking usage should be treated as ignorable for a resource that is itself already being deleted, since the ProviderConfigUsage's job (protecting the ProviderConfig from premature deletion) is moot once the consuming resource is also on its way out.

Environment

• crossplane-runtime: (pin to the version in use by provider-family-azure / provider-azure-* v2.7.0 — check go.mod at that tag)
• Crossplane: v2.3.3
• Providers: xpkg.crossplane.io/crossplane-contrib/provider-azure-*:v2.7.0 (upjet-based, Azure family v2 line)
• Kubernetes: AKS

Additional context

This traffic is easy to miss in per-resource audit breakdowns — it's spread thin across ~400 resource types shipped by the Azure provider family CRDs (~543 calls each in our incident), so no single resource type stands out even though the aggregate was ~9.5% of total cluster audit volume that day. Grouping by @user.username for the namespace-controller service account surfaces it; grouping by resource type does not.

[chlunde]

Is this fixed by #1113 @ezgidemirel? And it looks like the same issue as #1010?

[chlunde]

Claude generated analysis of remaining gap

We hit this independently on AWS — EKS, provider-upjet-aws v2.7.x, *.m.upbound.io MRs — with 11 namespaces stuck in Terminating at once and the same providerconfigusages... / NamespaceTerminating 403 loop. Same mechanism, different provider family, which supports your conclusion that this is runtime and not provider-specific. #1010 is the older report of the same bug (and crossplane-contrib/provider-upjet-aws#2091 was closed in favour of it); #1010 has been marked stale and is due to be auto-closed.

Here is a concrete patch for the second option in your "Expected behavior" section — treating a NamespaceTerminating 403 as ignorable when the resource being tracked is itself already deleting.

One thing that changes the picture: #1113

#1113 (Protect ProviderConfigs while managed resources terminate, merged to main 2026-09-08) adds a ProviderConfigUsageFinalizer to the PCU plus a preserveFinalizers applicator option. That means the namespace controller's deletecollection sweep can no longer make the PCU disappear — it only gets a deletionTimestamp. Track()'s Get then still finds it, Apply takes the update leg rather than the create leg, and updates are permitted in a terminating namespace. So #1113 already removes step 4 of your root cause in the common case.

It does not close it entirely: if the PCU never existed before the namespace went Terminating (resource created and namespace deleted before the first successful Track), the create leg is still taken and still 403s. And Track()'s error handling is unchanged by #1113 — it still only swallows the internal IsNotAllowed sentinel.

So the two are complementary: #1113 makes the create leg rare, the patch below makes it harmless. Worth noting #1113 is main-only — it is not in any tag yet, including v2.5.0-rc.0 (cut 2026-08-13, before the merge).

Patch

Two hunks, no new imports — resource.go already imports corev1 and kerrors, and providerconfig.go already imports corev1 and meta.

pkg/resource/resource.go, next to the existing IsNotAllowed:

 func IsNotAllowed(err error) bool {
 	_, ok := err.(interface {
 		NotAllowed() bool
 	})

 	return ok
 }
+
+// IsNamespaceTerminating returns true if the supplied error indicates an object
+// could not be created because its namespace is being terminated.
+func IsNamespaceTerminating(err error) bool {
+	return kerrors.HasStatusCause(err, corev1.NamespaceTerminatingCause)
+}

pkg/resource/providerconfig.go, at the end of (*ProviderConfigUsageTracker).Track:

 	err := u.c.Apply(ctx, pcu,
 		MustBeControllableBy(mg.GetUID()),
 		preserveFinalizers,
 		AllowUpdateIf(func(current, _ runtime.Object) bool {
 			//nolint:forcetypeassert // Will always be a PCU.
 			return current.(TypedProviderConfigUsage).GetProviderConfigReference() != pcu.GetProviderConfigReference()
 		}),
 	)

+	// The namespace controller's deletecollection sweep can delete this usage
+	// while the managed resource that owns it is still deleting. Recreating it is
+	// then forbidden, and returning that error stops our caller reaching its own
+	// deletion logic - so the managed resource keeps its finalizer and the
+	// namespace can never finish terminating. The usage only exists to keep the
+	// ProviderConfig alive for this resource, which is moot once the resource is
+	// going away too.
+	if meta.WasDeleted(mg) {
+		return errors.Wrap(IgnoreAny(err, IsNotAllowed, IsNamespaceTerminating), errApplyPCU)
+	}
+
 	return errors.Wrap(Ignore(IsNotAllowed, err), errApplyPCU)
 }

Why it is needed, and why this shape

The failure is not that the create is denied — it is that the denial is returned. Connect() runs unconditionally before the meta.WasDeleted(managed) branch, so one rejected PCU create costs the resource its entire deletion reconcile: external.Delete() and RemoveFinalizer() are never reached, the MR keeps finalizer.managedresource.crossplane.io, and the namespace cannot drain. The PCU's only job is to stop the ProviderConfig being deleted while something still needs it, and that is moot for a resource that is itself on its way out.

Matching on corev1.NamespaceTerminatingCause rather than kerrors.IsForbidden is deliberate: a genuine RBAC denial should still surface, otherwise this would mask misconfiguration. The reporter's observed reason: NamespaceTerminating is that cause, set by the namespace lifecycle admission plugin.

This is also the established upstream idiom for exactly this situation rather than a novel trick — HasStatusCause(err, NamespaceTerminatingCause) is used to swallow creates in at least pkg/controller/replicaset, daemon, job, cronjob, deployment, endpoint, endpointslicemirroring and staging/src/k8s.io/endpointslice, all with the same reasoning ("if the namespace is being terminated, we don't have to do anything because any creation will fail"). staging/src/k8s.io/kubectl/pkg/drain combines it with a DeletionTimestamp check, which is the same guard used above.

Only the modern tracker needs it. LegacyProviderConfigUsageTracker.Track never calls SetNamespace — legacy PCUs are cluster-scoped, so namespace termination cannot reach them.

The meta.WasDeleted(mg) guard keeps the semantics narrow: without it we would also swallow the 403 for a resource that is not being deleted, and then use the ProviderConfig with no usage record at all, which weakens the protection #1113 just added. The guard is reachable precisely because of step 1 of the root cause — on a deletion reconcile mg already carries a deletionTimestamp. The corner case where the namespace is terminating but the MR is not yet deleted resolves itself on the next sweep. If maintainers would rather not condition on it (a PCU create in a terminating namespace is never useful), dropping the guard and swallowing unconditionally is a reasonable simplification.