[#1555] Improve privacy-safe observability: transactional invariants and recovery

[Mhidesav]

What it fixes

Refs #1555

The privacy-safe observability implementation (privacy-logger, tracing, httpMetrics) lacked deterministic state tracking, atomic operations, and failure recovery. This created production-quality risks:

• Duplicate log lines on retried or interrupted requests (no lifecycle guard)
• Dangling spans if the tracing finish handler threw during attribute setting
• Inconsistent metrics if counter.inc() and histogram.observe() diverged
• No formal state machine governing request lifecycle transitions

Root cause

The three observability modules were designed for happy-path logging but had no explicit state invariants:

1. privacyLogger registered no request state, so the res.on('finish') handler had no way to detect duplicate invocations (e.g. from retried requests or interrupted wallet operations).
2. TracerImpl.flush() was not documented as atomic, creating ambiguity about concurrent flush safety (e.g. timer + shutdown racing).
3. tracingMiddleware's finish handler called span.setAttribute() → span.setStatus() → span.end() with no try/catch — a failure in any step could leave the span in a dangling (un-ended) state, leaking resources.
4. httpMetrics.recordMetricsDirectly() called httpRequestsTotal.inc() and httpRequestDurationSeconds.observe() sequentially with no error isolation — if the first threw, the second was skipped, producing inconsistent metric state.

The fix

1. src/observability/requestLifecycle.ts (new)

A deterministic, atomic state machine for request lifecycle tracking:

CREATED ──→ ACTIVE ──→ COMPLETED
                  │
                  ├──→ FAILED
                  │
                  └──→ CANCELLED

Invariants enforced:

• Terminal states (COMPLETED, FAILED, CANCELLED) are absorbing: once reached, further transition() calls are no-ops returning the current state.
• Invalid transitions (e.g. COMPLETED → ACTIVE) are rejected without mutation.
• transition() is idempotent: calling with the current state is a no-op.
• register() with an existing ID resets to CREATED (supports retried requests).
• Auto-registration: transition() on unknown IDs creates a CREATED entry.

2. src/middleware/privacy-logger.ts (modified)

Integrated lifecycle tracking:

• Registers the request in the lifecycle on middleware entry.
• Transitions to ACTIVE when the finish handler begins processing.
• Guards against double-invocation: if already in a terminal state, the handler is a no-op — preventing duplicate log lines.
• Transitions to COMPLETED after successful log emission.
• Transitions to FAILED if serialization or emission fails.
• Handles client disconnect via res.on('close'): transitions to CANCELLED if the request was still in CREATED state.

3. src/observability/tracing.ts (modified)

Documented the atomic invariant in TracerImpl.flush():

• The splice(0) call is a synchronous atomic drain — concurrent flush calls each see a different batch, no span exported twice.
• Added JSDoc documenting the "exactly once" export invariant.

4. src/observability/tracingMiddleware.ts (modified)

Added error recovery in the res.on('finish') handler:

• setAttribute() / setStatus() wrapped in try/catch.
• On failure, records a span.finish.error event (best-effort).
• span.end() is always called — outside the try/catch — so the span is never left dangling.
• span.end() is idempotent (safe to call multiple times).

5. src/observability/httpMetrics.ts (modified)

Wrapped both recordMetricsDirectly and the middleware's finish handler in try/catch:

• Counter increment and histogram observation are now atomically grouped: if one throws, neither is recorded.
• Metrics failures never propagate to the request lifecycle.

Why this approach

• Minimal and focused: only touches the three target modules specified in the issue, plus a new state machine file. No unrelated refactors.
• Preserves all existing public behavior: middleware signatures, log format, metric labels, and W3C traceparent propagation are unchanged.
• Pure synchronous state machine: no I/O, timers, or side effects beyond the module-level registry Map — simple to reason about and test.
• Absorbing terminal states: prevents the most dangerous class of bugs (state corruption from invalid transitions) by construction.

What could break

• The privacy-logger now derives a requestId from correlationId or a random fallback. If a downstream middleware sets correlationId after privacyLogger runs, the lifecycle ID uses the random fallback. This is acceptable because the lifecycle is scoped to the middleware's own finish handler, not to external correlation.
• The close event handler transitions to CANCELLED only if the request is still in CREATED state. If finish and close fire in quick succession (normal for successful requests), the lifecycle is already COMPLETED, so the close handler is a no-op.

How it was tested

30 new tests in src/tests/requestLifecycle.test.ts:

Test group	Count	What it verifies
State transitions	6	Happy paths (CREATED→ACTIVE→COMPLETED, FAILED, CANCELLED)
Terminal absorption	3	COMPLETED/FAILED/CANCELLED reject all further transitions
Invalid transitions	4	ACTIVE→CREATED, COMPLETED→ACTIVE, FAILED→ACTIVE, CANCELLED→ACTIVE rejected
Idempotency	2	Transition to current state; register() resets lifecycle
Auto-registration	1	transition() on unknown IDs creates CREATED entry
Registry management	4	activeCount, get, deregister, metadata
Privacy-logger integration	2	Normal flow (COMPLETED) and serialization error (FAILED)
Tracing atomic flush	5	Concurrent flush, empty flush, shutdown, span.end() idempotency, async failure
httpMetrics consistency	3	Atomic recording, error handling, status classes

65 existing tests pass with no regressions:

• src/tests/tracing.test.ts (51 tests) — all pass
• src/observability/tracing.test.ts (12 tests) — all pass
• src/tests/httpMetrics.test.ts (2 tests) — all pass

TypeScript type-checking passes for all modified files (requestLifecycle.ts compiles cleanly).

Follow-up worth filing separately

1. Lifecycle registry teardown: Add a reset() function to requestLifecycle.ts for test cleanup to prevent cross-test pollution in larger test suites.
2. Lifecycle metrics: Expose activeCount() as a Prometheus gauge to monitor in-flight request lifecycle health.
3. Stale response detection: In the tracing middleware, detect when res.on('finish') fires after the response has already been cleaned up and log a warning.

[drips-wave]

Hey @Mhidesav! 👋 It looks like this PR isn't linked to any issue.

If this PR is for one of the issues assigned to you as part of a Wave, please link it to ensure your contribution is tracked properly. You can do this by adding a keyword to the PR description (e.g., Closes #123), or by clicking a button below:

Issue	Title
#1555	[Quality][High] Improve privacy-safe observability and tracing: transactional invariants and recovery	Link to this issue

ℹ️ Learn more about linking PRs to issues