feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption

[jsonbailey]

Summary

• Per-execution runId: Every tracker now includes a unique runId (UUID) in all track event payloads, enabling billing isolation per execution
• At-most-once semantics: Each metric type (duration, tokens, success/error, feedback, time-to-first-token) can only be tracked once per tracker instance — subsequent calls are silently dropped with a log warning
• create_tracker() factory on config objects: AICompletionConfig, AIAgentConfig, and AIJudgeConfig now carry an optional create_tracker callable that returns a fresh LDAIConfigTracker with a new runId each time it's called. Set to None when the config is disabled.
• Per-invocation trackers in managed classes: ManagedModel.invoke(), ManagedAgent.run(), and Judge.evaluate() now call create_tracker() at the start of each invocation to get a fresh tracker, fixing the multi-turn tracking issue where at-most-once guards blocked metrics from second+ invocations
• resumption_token property on tracker: URL-safe Base64-encoded (no padding) JSON string containing {runId, configKey, variationKey, version} for cross-process tracker reconstruction
• LDAIClient.create_tracker(token, context): Reconstructs a tracker from a resumption token for deferred feedback scenarios. Validates required fields and raises ValueError for invalid tokens.

Test plan

• Enabled config has create_tracker callable; disabled config has None
• Each create_tracker() call returns a new tracker with a distinct runId
• Factory closure captures correct flag metadata (configKey, variationKey, version, modelName, providerName)
• ManagedAgent.run() uses create_tracker() when available, falls back to stored tracker
• Resumption token round-trip encode/decode preserves all fields
• Resumption token has no base64 padding characters
• create_tracker(token, context) reconstructs tracker with original runId and empty model/provider
• Invalid base64, invalid JSON, and missing required fields all raise ValueError
• All 137 existing + new tests pass with no regressions

🤖 Generated with Claude Code

---

Note

High Risk
High risk because it refactors core tracking APIs across configs, managed wrappers, and both LangChain/OpenAI graph runners, changing tracker lifecycles and event payloads. Incorrect factory usage or caching could break metrics emission or correlation across runs.

Overview
Moves tracking from a stored tracker instance to a per-invocation create_tracker() factory across configs, managed wrappers (ManagedModel, ManagedAgent, ManagedAgentGraph), judges, and both LangChain/OpenAI agent-graph runners, ensuring each execution gets a fresh tracker.

Adds a per-execution runId to all LDAIConfigTracker events, introduces at-most-once guards for key metrics (duration, tokens, success/error, feedback, TTF), and implements cross-process tracker resumption via LDAIConfigTracker.resumption_token + LDAIClient.create_tracker(token, context).

Updates agent-graph tracking to cache per-node tracker instances during a run (so tool calls/durations/tokens share the same runId) and adjusts tests to validate factory call counts, runId consistency, and resumption-token behavior.

^{Reviewed by Cursor Bugbot for commit d721142. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d721142. Configure here.}

[cursor]

Redundant disabled classmethod overrides in config subclasses

Low Severity

AICompletionConfigDefault.disabled(), AIAgentConfigDefault.disabled(), and AIJudgeConfigDefault.disabled() all override the base AIConfigDefault.disabled() with identical logic (return cls(enabled=False)). The base class already uses cls, so calling disabled() on any subclass correctly returns an instance of that subclass. The overrides only add narrower return-type annotations, which could instead be achieved with typing.Self.

Additional Locations (2)

• packages/sdk/server-ai/src/ldai/models.py#L255-L258
• packages/sdk/server-ai/src/ldai/models.py#L305-L308

 

^{Reviewed by Cursor Bugbot for commit d721142. Configure here.}