fix(agents): emit terminal stream END only after finalize_turn

[daryllimyt]

Summary

Backend root fix for ENG-1554: the final assistant message streams in fully, then vanishes at end of turn until reload.

The race: the executor loopback persists the final assistant rows (tagged curr_run_id) and then emits the stream END token, but finalize_turn — which clears curr_run_id and thereby makes the rows visible to GET /sessions/{id}/vercel — only runs later in the workflow finally. A client refetch triggered by END lands in that window and gets a snapshot with the just-finished turn hidden.

This PR establishes the invariant that a client-observable terminal END implies the canonical DB snapshot is publishable: persist terminal state → finalize_turn (compare-and-clear by run id) → emit END to the captured stream id.

The frontend defense-in-depth layer (content-aware adopt guard) is #3119; regression origin and full RCA are in the Linear issue.

Changes

• The workflow now owns Redis stream completion. The executor loopback never calls done() on Redis; it closes only external sinks via done_external().
• At an approval pause, the workflow persists approval requests and then emits the pause END. curr_run_id remains set because the turn is still active, and a resumed turn uses the possibly rotated stream id.
• At a terminal boundary, the workflow captures the active stream id, runs _finalize_turn, and then calls the existing idempotent emit_session_done activity. An ambiguous retry may append a duplicate END, which clients already tolerate. If finalization exhausts its retries, END is still attempted so the client does not hang indefinitely.
• Error events still stream inline immediately, but their END is workflow-owned. Cancellation markers are persisted before terminal END; the early cancellation notice keeps its existing timing.

Temporal compatibility

• FINALIZE_TURN_PATCH remains gated with workflow.patched(...). Histories without that marker must not schedule the newer finalize_turn activity.
• The approval-stream v2 behavior is now unconditional, but APPROVAL_STREAM_V2_PATCH remains at its original command boundary as workflow.deprecate_patch(...). This is a replay bridge for histories that recorded the v2 marker and is covered by a real history-replay regression test. Keep the bridge until those histories have drained.
• The bridge only covers v2-marked histories. Pre-v2 histories that already advanced past an approval pause did not record the now-unconditional emit_session_done command and are not replay-compatible with that boundary; verify that no such executions remain RUNNING before rollout.
• Pre-change histories that already completed without a workflow-owned terminal END are likewise not strict-replay-compatible with the new terminal activity. Normal client refresh rebuilds from canonical DB/Redis state, but those closed executions should not be reset or replayed against this worker version.
• REMINT_SCOPE_TOKENS_PATCH remains gated because approval waits are unbounded, so affected histories cannot be assumed to have drained.
• The intermediate durable-agent-terminal-end-after-finalize-v1 marker from an earlier revision of this PR was removed. It never reached main, so there are no mainline histories carrying it to preserve.

Tests

• Workflow-level DB regression: the END activity observes curr_run_id IS NULL and the final rows through list_messages(include_active=False).
• Real Temporal history replay for a suspended workflow carrying durable-agent-approval-stream-v2, plus existing suspended and completed replay coverage.
• Cancellation ordering, approval-pause persistence, rotated continuation stream ids, and finalize-failure behavior.
• Loopback and sandbox tests verify Redis END is workflow-owned while external sinks still close executor-side.
• Focused Temporal tests, Ruff, formatting, and BasedPyright pass locally.

---

Summary by cubic

Fixes ENG-1554: emit terminal END only after finalize_turn so the final assistant message stays visible at end-of-turn. The workflow now owns stream completion; the executor only closes external sinks.

• Bug Fixes
  • Terminal END runs inside finalize_turn_activity when FINALIZE_TURN_WITH_END_PATCH is present: capture the active stream id, compare-and-clear by run id, then emit END. If a legacy worker runs this activity (returns no result), the workflow falls back to emit_session_done. If the combined finalize activity fails, we do not emit END.
  • Legacy histories remain compatible: if only FINALIZE_TURN_PATCH is recorded, the workflow finalizes (DB-only) and then emits END separately.
  • Executor loopback never calls done() on Redis; it closes only external sinks. Approval pauses only close external sinks; deduplicated. Errors still stream inline immediately.
  • emit_session_error and emit_session_cancelled persist their frames (and stream notices when needed) but never call done(); the workflow emits terminal END after finalize. Cancellations persist the marker before END.
  • Approvals: after persisting approvals, the workflow closes the pause stream; rotated stream ids are respected at resume and terminal END. APPROVAL_STREAM_V2_PATCH is retained via workflow.deprecate_patch(...); the executor’s defer_done_on_approval flag is removed.
  • Tests: DB-backed checks that END observes publishable final history; legacy-worker fallback; combined-finalize failure (no END); approval pause, cancellation ordering, and rotated stream ids; replay of the approval-stream v2 marker; and a fixture that closes the shared Redis client per durable workflow test.

^{Written for commit fe05cb3. Summary will update on new commits.}

[zeropath-ai]

✅ No security or compliance issues detected. Reviewed everything up to fe05cb3.

Security Overview

• 🔎 Scanned files: 13 changed file(s)
• 🔗 Scan Link: https://zeropath.com/app/repositories/00dffd6c-8834-4dc9-b6d8-b44cd1622986?scanId=0047efc9-5c73-4b21-baaf-f3524f1f4a9d&codeScanTypes=PrScan&tab=issues

Detected Code Changes

Change Type

Relevant files

Enhancement

► packages/tracecat-ee/tracecat_ee/agent/activities.py     Improve session end/terminal handling strings and flow comments ► packages/tracecat-ee/tracecat_ee/agent/workflows/durable.py     Introduce FinalizeTurnPatch handling, add FinalizeTurnWithEnd patch, extend finalize_turn flow, and add terminal done emission logic ► tests/temporal/test_durable_agent_workflow.py     Update tests to accommodate new finalize_turn behavior, terminal done emission, and patch deprecation/replay logic ► tests/unit/test_agent_activities.py     Extend tests for emit_session_error and terminal handling with done state ► tests/unit/test_agent_executor_loopback.py     Adjust tests for terminal end/done behavior and handler state expectations

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ee9e4e067c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jordan-umusu]

lgtm

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a69290df84

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[jordan-umusu]

LGTM on the current impl., but gentle pushback on the shape:

I think terminal END emission should be folded into the existing finalize_turn_activity, with AgentSessionService.finalize_turn enforcing the following invariant:

When finalize_turn returns, this run no longer owns the session’s active DB pointers and its captured Redis stream contains END.

The method should clear and commit the DB pointers, then emit END. If either step fails, it raises and Temporal retries the entire activity. The DB operation is
idempotent, and duplicate END frames are harmless (the current impl. can already do this).

[daryllimyt]

@jordan-umusu Addressed in fe05cb3.

• AgentSessionService.finalize_turn now compare-and-clears and commits the active DB pointers, then emits END to the captured Redis stream. A Redis failure propagates so Temporal retries the whole activity.
• A new Temporal patch preserves the pre-v1 and v1 replay command shapes, with a mixed-worker fallback for rolling deploys.
• Verified: 127 unit tests passed (2 skipped); the durable-agent Temporal module passed 20 tests (4 skipped); Ruff, Ruff formatting, BasedPyright, diff checks, secret scanning, and pre-commit all passed.

Could you take another look?

[jordan-umusu]

LGTM

[jordan-umusu]

nit: remove the branching, this likely doesnt buy much