Skip to content

feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption #133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jsonbailey merged 31 commits into from
Apr 21, 2026
Merged

feat!: Add per-execution runId, at-most-once tracking, and cross-process tracker resumption #133

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
2b997fa
feat!: Add per-execution runId and at-most-once event tracking
jsonbailey Apr 15, 2026
211ead4
feat!: Add per-execution runId, at-most-once tracking, and cross-proc…
jsonbailey Apr 15, 2026
6237d6c
refactor: Move UUID generation from tracker constructor to factory cl…
jsonbailey Apr 16, 2026
d895e64
fix: Fix CI lint errors and add run_id to provider tests
jsonbailey Apr 16, 2026
7839104
refactor: Use LDAIMetricSummary fields as at-most-once guards
jsonbailey Apr 16, 2026
df722f9
refactor: Reorder LDAIConfigTracker __init__ params to match spec
jsonbailey Apr 16, 2026
a6e9612
refactor: Move context before model/provider params, fix resumption t…
jsonbailey Apr 16, 2026
59c574e
chore: Include track data in at-most-once warning logs
jsonbailey Apr 16, 2026
6bf91fa
feat: Add from_resumption_token classmethod to LDAIConfigTracker
jsonbailey Apr 16, 2026
ba5421a
fix: Omit variationKey from track data when empty
jsonbailey Apr 16, 2026
4c0451b
feat: Always set create_tracker as callable that returns a tracker
jsonbailey Apr 17, 2026
82cb40a
feat!: Remove config.tracker field — use create_tracker() instead
jsonbailey Apr 17, 2026
7050ab0
feat: Include graphKey in resumption token when set
jsonbailey Apr 17, 2026
31dcc8f
feat!: Remove tracker param from Judge and ManagedAgentGraph
jsonbailey Apr 17, 2026
08da63a
chore: Use ldai.log instead of logging module in tracker
jsonbailey Apr 17, 2026
ae5d752
fix: Update provider packages for tracker factory pattern
jsonbailey Apr 17, 2026
fc814c6
feat!: Replace AgentGraphDefinition.get_tracker() with create_tracker…
jsonbailey Apr 17, 2026
5313ce5
fix: Cache node trackers per execution to preserve runId correlation
jsonbailey Apr 17, 2026
04f14eb
fix: Create graph tracker once per run, not twice
jsonbailey Apr 17, 2026
dd44577
feat!: Return Result from from_resumption_token instead of raising
jsonbailey Apr 17, 2026
84a1ab1
fix: Type create_tracker as Optional to fix mypy errors
jsonbailey Apr 17, 2026
a41c7e3
feat!: Remove public disabled() classmethod from AIConfigDefault
jsonbailey Apr 17, 2026
edd1690
fix: Guard against None tracker in Judge.evaluate()
jsonbailey Apr 20, 2026
2fc7151
fix: Address remaining PR review feedback
jsonbailey Apr 20, 2026
609451e
refactor!: Make create_tracker required on AIConfig (no default)
jsonbailey Apr 20, 2026
db25c8f
feat: Add disabled() classmethod to all AIConfigDefault variants
jsonbailey Apr 20, 2026
0aeb164
refactor: Use disabled() classmethod for fallback defaults in client
jsonbailey Apr 20, 2026
d721142
chore: Remove unused DEFAULT_FALSE constant from agent_graph
jsonbailey Apr 20, 2026
88c18b9
refactor: Use typing.Self for disabled() return type, remove redundan…
jsonbailey Apr 21, 2026
bed2034
fix: Import Self from typing_extensions for Python 3.10 compat
jsonbailey Apr 21, 2026
35aef5e
fix: Use truthy check for graph_key in __get_track_data
jsonbailey Apr 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/ai-providers/server-ai-langchain/src/ldai_langchain/langgraph_callback_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ def flush(self, graph: AgentGraphDefinition) -> None:
node = graph.get_node(node_key)
if not node:
continue
config_tracker = node.get_config().tracker
config_tracker = node.get_config().create_tracker()
Comment thread
cursor[bot] marked this conversation as resolved.
if not config_tracker:
continue

Expand Down
2 changes: 0 additions & 2 deletions packages/ai-providers/server-ai-langchain/tests/test_langchain_provider.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -536,7 +536,6 @@ def sync_tool(x: str = '') -> str:
),
provider=ProviderConfig(name='openai'),
instructions='',
tracker=MagicMock(),
)
tools = build_structured_tools(cfg, {'my_tool': sync_tool})
assert len(tools) == 1
Expand All @@ -559,7 +558,6 @@ async def async_tool(x: str = '') -> str:
),
provider=ProviderConfig(name='openai'),
instructions='',
tracker=MagicMock(),
)
tools = build_structured_tools(cfg, {'my_tool': async_tool})
assert len(tools) == 1
Expand Down
1 change: 0 additions & 1 deletion packages/ai-providers/server-ai-langchain/tests/test_langgraph_agent_graph_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ def _make_graph(enabled: bool = True) -> AgentGraphDefinition:
model=ModelConfig(name='gpt-4'),
provider=ProviderConfig(name='openai'),
instructions='You are a helpful assistant.',
tracker=MagicMock(),
)
graph_config = AIAgentGraphConfig(
key='test-graph',
Expand Down
7 changes: 4 additions & 3 deletions packages/ai-providers/server-ai-langchain/tests/test_langgraph_callback_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ def _make_graph(mock_ld_client: MagicMock, node_key: str = 'root-agent', graph_k
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
graph_key=graph_key,
)
graph_tracker = AIGraphTracker(
Expand All @@ -50,7 +51,7 @@ def _make_graph(mock_ld_client: MagicMock, node_key: str = 'root-agent', graph_k
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='Be helpful.',
tracker=node_tracker,
create_tracker=lambda: node_tracker,
)
graph_config = AIAgentGraphConfig(
key=graph_key,
Expand Down Expand Up @@ -402,14 +403,15 @@ def test_flush_with_no_graph_key_on_node_tracker():
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
)
node_config = AIAgentConfig(
key='root-agent',
enabled=True,
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='Be helpful.',
tracker=node_tracker,
create_tracker=lambda: node_tracker,
)
graph_config = AIAgentGraphConfig(
key='test-graph',
Expand Down Expand Up @@ -463,7 +465,6 @@ def test_flush_skips_node_without_tracker():
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='',
tracker=None,
)
graph_config = AIAgentGraphConfig(
key='g', root_config_key='no-track', edges=[], enabled=True
Expand Down
23 changes: 14 additions & 9 deletions packages/ai-providers/server-ai-langchain/tests/test_tracking_langgraph.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ def _make_graph(
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
graph_key=graph_key,
)

Expand All @@ -68,7 +69,7 @@ def _make_graph(
model=ModelConfig(name='gpt-4', parameters={'tools': tool_defs} if tool_defs else {}),
provider=ProviderConfig(name='openai'),
instructions='You are a helpful assistant.',
tracker=node_tracker,
create_tracker=lambda: node_tracker,
)

graph_config = AIAgentGraphConfig(
Expand Down Expand Up @@ -142,6 +143,7 @@ def _make_two_node_graph(mock_ld_client: MagicMock) -> 'AgentGraphDefinition':
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
graph_key='two-node-graph',
)
child_tracker = LDAIConfigTracker(
Expand All @@ -152,6 +154,7 @@ def _make_two_node_graph(mock_ld_client: MagicMock) -> 'AgentGraphDefinition':
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
graph_key='two-node-graph',
)
graph_tracker = AIGraphTracker(
Expand All @@ -168,15 +171,15 @@ def _make_two_node_graph(mock_ld_client: MagicMock) -> 'AgentGraphDefinition':
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You are root.',
tracker=root_tracker,
create_tracker=lambda: root_tracker,
)
child_config = AIAgentConfig(
key='child-agent',
enabled=True,
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You are child.',
tracker=child_tracker,
create_tracker=lambda: child_tracker,
)

edge = Edge(key='root-to-child', source_config='root-agent', target_config='child-agent')
Expand Down Expand Up @@ -514,6 +517,7 @@ def _make_multi_child_graph(mock_ld_client: MagicMock) -> 'AgentGraphDefinition'
def _node_tracker(key: str) -> LDAIConfigTracker:
return LDAIConfigTracker(
ld_client=mock_ld_client,
run_id='test-run-id',
variation_key='test-variation',
config_key=key,
version=1,
Expand All @@ -538,23 +542,23 @@ def _node_tracker(key: str) -> LDAIConfigTracker:
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='Route to the appropriate specialist agent.',
tracker=_node_tracker('orchestrator'),
create_tracker=lambda: _node_tracker('orchestrator'),
),
'agent-a': AIAgentConfig(
key='agent-a',
enabled=True,
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You handle topic A.',
tracker=_node_tracker('agent-a'),
create_tracker=lambda: _node_tracker('agent-a'),
),
'agent-b': AIAgentConfig(
key='agent-b',
enabled=True,
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You handle topic B.',
tracker=_node_tracker('agent-b'),
create_tracker=lambda: _node_tracker('agent-b'),
),
}

Expand Down Expand Up @@ -625,6 +629,7 @@ def _make_multi_child_graph_with_tools(mock_ld_client: MagicMock, tool_names: li
def _node_tracker(key: str) -> LDAIConfigTracker:
return LDAIConfigTracker(
ld_client=mock_ld_client,
run_id='test-run-id',
variation_key='test-variation',
config_key=key,
version=1,
Expand All @@ -650,23 +655,23 @@ def _node_tracker(key: str) -> LDAIConfigTracker:
model=ModelConfig(name='gpt-4', parameters={'tools': tool_defs}),
provider=ProviderConfig(name='openai'),
instructions='Route to a specialist after gathering info.',
tracker=_node_tracker('orchestrator'),
create_tracker=lambda: _node_tracker('orchestrator'),
),
'agent-a': AIAgentConfig(
key='agent-a',
enabled=True,
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You handle topic A.',
tracker=_node_tracker('agent-a'),
create_tracker=lambda: _node_tracker('agent-a'),
),
'agent-b': AIAgentConfig(
key='agent-b',
enabled=True,
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You handle topic B.',
tracker=_node_tracker('agent-b'),
create_tracker=lambda: _node_tracker('agent-b'),
),
}

Expand Down
6 changes: 3 additions & 3 deletions packages/ai-providers/server-ai-openai/src/ldai_openai/openai_agent_graph_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,7 @@ def _build_agents(self, path: List[str], state: _RunState) -> Any:

def build_node(node: AgentGraphNode, ctx: dict) -> Any:
node_config = node.get_config()
config_tracker = node_config.tracker
config_tracker = node_config.create_tracker()
model = node_config.model

if not model:
Expand Down Expand Up @@ -266,7 +266,7 @@ def _flush_final_segment(
node = self._graph.get_node(state.last_node_key)
if node is None:
return
config_tracker = node.get_config().tracker
config_tracker = node.get_config().create_tracker()
if config_tracker is None:
return

Expand Down Expand Up @@ -296,6 +296,6 @@ def _track_tool_calls(self, result: Any) -> None:
node = self._graph.get_node(agent_key)
if node is None:
continue
config_tracker = node.get_config().tracker
config_tracker = node.get_config().create_tracker()
Comment thread
cursor[bot] marked this conversation as resolved.
Outdated
if config_tracker is not None:
config_tracker.track_tool_call(tool_name)
5 changes: 3 additions & 2 deletions packages/ai-providers/server-ai-openai/tests/test_openai_agent_graph_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,14 @@

def _make_graph(enabled: bool = True) -> AgentGraphDefinition:
"""Build a minimal single-node AgentGraphDefinition for testing."""
node_tracker = MagicMock()
root_config = AIAgentConfig(
key='root-agent',
enabled=enabled,
model=ModelConfig(name='gpt-4'),
provider=ProviderConfig(name='openai'),
instructions='You are a helpful assistant.',
tracker=MagicMock(),
create_tracker=lambda: node_tracker,
)
graph_config = AIAgentGraphConfig(
key='test-graph',
Expand Down Expand Up @@ -136,7 +137,7 @@ async def test_openai_agent_graph_runner_run_success():
tracker.track_path.assert_called_once()
tracker.track_latency.assert_called_once()

root_tracker = graph.get_node('root-agent').get_config().tracker
root_tracker = graph.get_node('root-agent').get_config().create_tracker()
root_tracker.track_duration.assert_called_once()
root_tracker.track_tokens.assert_called_once()
root_tracker.track_success.assert_called_once()
9 changes: 6 additions & 3 deletions packages/ai-providers/server-ai-openai/tests/test_tracking_openai_agents.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ def _make_graph(
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
graph_key=graph_key,
)

Expand All @@ -63,7 +64,7 @@ def _make_graph(
model=ModelConfig(name='gpt-4', parameters={'tools': tool_defs} if tool_defs else {}),
provider=ProviderConfig(name='openai'),
instructions='You are a helpful assistant.',
tracker=node_tracker,
create_tracker=lambda: node_tracker,
)

graph_config = AIAgentGraphConfig(
Expand Down Expand Up @@ -179,6 +180,7 @@ def _make_two_node_graph(mock_ld_client: MagicMock) -> AgentGraphDefinition:
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
graph_key='two-node-graph',
)
child_tracker = LDAIConfigTracker(
Expand All @@ -189,6 +191,7 @@ def _make_two_node_graph(mock_ld_client: MagicMock) -> AgentGraphDefinition:
model_name='gpt-4',
provider_name='openai',
context=context,
run_id='test-run-id',
graph_key='two-node-graph',
)
graph_tracker = AIGraphTracker(
Expand All @@ -205,15 +208,15 @@ def _make_two_node_graph(mock_ld_client: MagicMock) -> AgentGraphDefinition:
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You are root.',
tracker=root_tracker,
create_tracker=lambda: root_tracker,
)
child_config = AIAgentConfig(
key='child-agent',
enabled=True,
model=ModelConfig(name='gpt-4', parameters={}),
provider=ProviderConfig(name='openai'),
instructions='You are child.',
tracker=child_tracker,
create_tracker=lambda: child_tracker,
)

edge = Edge(key='root-to-child', source_config='root-agent', target_config='child-agent')
Expand Down
Loading
Loading