fix(hosting): keep the API alive when a background service throws

[JabbaKadabra]

Closes #522.

The failure

.NET's default HostOptions.BackgroundServiceExceptionBehavior is StopHost: an unhandled exception in any one BackgroundService stops the whole host and the process exits with code 0.

That zero exit is the trap. It is indistinguishable from an intentional shutdown, so no restart policy treats it as a failure — the container just stays down — and the Critical log line explaining it never gets persisted, because the process is already going away. In #522 a licensed e2e api container passed its healthcheck, then vanished ~35 s into the run while the identical api-free stayed healthy; every subsequent request hit nginx's error page and the setup spec died parsing HTML as JSON, with no evidence of which service threw.

What changed

1. A faulted background loop no longer kills the host. Both process hosts (Proxytrace.Api, Proxytrace.Proxy.Api) now call AddResilientBackgroundServices(), a new extension in Proxytrace.Common/Hosting/, which sets the behaviour to Ignore. The faulted loop stops; everything else — including the HTTP surface — keeps serving. The framework logs the fault at Error, which is exactly the level ErrorLogChannelLoggerProvider captures, so the crash lands in the in-product error log instead of dying with the process.

The trade is deliberate: losing one background loop is a degraded feature, losing the host is a total outage. Ignore keeps the host alive — it does not restart the loop, so a loop that wants to survive its own transient failures still has to catch them itself (the shape ErrorLogWriter and LicenseCheckService.SafeRunCheckAsync already use).

Startup-critical work is unaffected, and this is load-bearing: the option only covers BackgroundService.ExecuteAsync. Schema initialization/migrations (DatabaseInitializationService), the secret/preview/tool backfills and the seeders are all plain IHostedService with their work in StartAsync, which still aborts startup when it throws. An API must not serve traffic against an unmigrated database, and that stays true.

2. e2e failures are diagnosable from the run page. The Capture e2e stack logs step now also tees a bounded 200-line tail of every service into the job log, one collapsed group per service. The per-service logs already reached the e2e-stack-logs artifact, but the artifact is not always reachable from wherever the run is triaged — which is why #522 could not name the throwing service at all.

Tests

Three new tests in Proxytrace.Common.Tests/Hosting/, building a real IHost because the behaviour under test is the host's own fault handling:

• the extension configures Ignore;
• a mid-run BackgroundService fault leaves the host running with it applied;
• the negative control — the same fault on the framework default stops the host, so the test would go red if the wiring were ever dropped.

Verified with the full backend suite (dotnet test Proxytrace.sln) — cross-cutting change: it touches Proxytrace.Common, both composition roots, and adds a package reference. All green except two pre-existing RedisIngestionStreamIntegrationTests failures caused by this environment having no Docker socket; their skip guard misses the throw and is filed separately as #526.

Docs

• docs/architecture.md — new section on the BackgroundService (loop, non-fatal) vs IHostedService (startup-critical, fatal) split and which to reach for.
• docs/testing.md — the job-log tails alongside the artifact.
• manual/admin/error-log.md — an operator-facing warning: a failed background service is recorded here, the deployment stays up, but the loop does not come back until the API is restarted.
• CHANGELOG.md under [Unreleased] → Fixed.

---

Generated by Claude Code