feat(agent): retry agent turn once on infra failure - #3145
Conversation
A worker loss (e.g. the recent agent-executor OOM kill) heartbeat-fails every in-flight run_agent_activity with maximum_attempts=1, so all affected sessions and their parent workflows fail outright even though a healthy worker was polling before the timeouts fired. Add a dedicated activity:agent_turn retry policy (maximum_attempts=2, same non-retryable error list, so only infrastructure failures retry) and use it at both run_agent_activity call sites. The shared activity:fail_fast policy and all other call sites are unchanged.
daryllimyt
added
enhancement
|
✅ No security or compliance issues detected. Reviewed everything up to dfa2ec6. Security Overview
Detected Code Changes
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dfa2ec624f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ), | ||
| heartbeat_timeout=timedelta(seconds=60), | ||
| retry_policy=RETRY_POLICIES["activity:fail_fast"], | ||
| retry_policy=RETRY_POLICIES["activity:agent_turn"], |
There was a problem hiding this comment.
When the first attempt fails late in a turn—especially on the configured start-to-close timeout—the retry reuses executor_input, including the original MCP and LLM JWTs. Both token types expire after TRACECAT__AGENT_SANDBOX_TIMEOUT + 60 seconds (tracecat/agent/tokens.py:174 and :366), while each activity attempt may run for the full sandbox timeout. Consequently, a retry after a start-to-close timeout begins with only about 60 seconds of token validity, and a retry after a late heartbeat timeout may start with an already-expired token, causing authenticated LLM/tool calls to fail instead of recovering. Mint tokens with enough lifetime for both attempts or refresh them per attempt before enabling this retry policy at both call sites.
Useful? React with 👍 / 👎.
1 participant
Why
When the agent-executor pod was OOM-killed on 2026-07-27, all ten in-flight
run_agent_activityexecutions heartbeat-timed-out withRETRY_STATE_MAXIMUM_ATTEMPTS_REACHEDand every affectedDurableAgentWorkflow(and parent workflow) failed — even though a replacement worker was polling 47 seconds before the first timeout fired. One retry would very likely have recovered all ten sessions.What
A dedicated
activity:agent_turnretry policy (maximum_attempts=2, sameNON_RETRYABLE_ERROR_TYPESlist) applied at bothrun_agent_activitycall sites inDurableAgentWorkflow. Application/validation errors still fail on the first attempt — only infrastructure failures (heartbeat timeout, worker loss, timeouts) get the second attempt. The sharedactivity:fail_fastpolicy and its ~17 other call sites in the DSL and EE workflows are unchanged, with a regression test pinning both policies.Reviewed in depth before shipping; retries trade certain failure for rare duplication artifacts:
sdk_session_id=None, so it starts an independent SDK session; the Tracecat session history then contains both attempts' transcripts chronologically merged.LoopbackHandler._persisted_line_uuidsstarts empty per attempt; no DB uniqueness constraint on the content UUID), so it does not protect across attempts — though the resumed-file offset means attempt 1's exact lines are normally not re-emitted.Rationale for shipping anyway: the retry only fires on worker loss (rare, infra-caused), and the status quo is guaranteed failure of the session and its parent workflow. Follow-ups that would close the gap: reload session state at activity start for retries (mirror the approval-resume path), hydrate the SDK session ID on first-turn retries, and cross-attempt line dedup.
Testing
tests/unit/test_retry_policies.py: pinsactivity:agent_turn(attempts=2, non-retryable list identity) andactivity:fail_fast(attempts=1); accessing the exact key guards the runtime-KeyError failure mode.ruff,basedpyright --warnings, pytest on touched files: clean.Summary by cubic
Add a one-time retry for agent turns to recover from infra failures like worker loss or heartbeat timeouts. This reduces failed sessions and parent workflows when a worker restarts mid-turn.
activity:agent_turnretry policy intracecat.dsl.common(maximum_attempts=2, sharesNON_RETRYABLE_ERROR_TYPESso app errors still fail fast).run_agent_activitycall sites inDurableAgentWorkflow;activity:fail_fastand other call sites remain unchanged.tests/unit/test_retry_policies.pyto pin both policies.Written for commit dfa2ec6. Summary will update on new commits.