fix(space): rate-limit task-lifecycle recovery — banner-Cancel respawn + partial-resume recompute - #2318
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Greptile SummaryFixes two rate-limit task-lifecycle edge cases: (A) after the cooldown banner's Cancel, the task was unrecoverable in-process because retryNow couldn't re-fire and the cross-restart sweep skipped live idle sessions; (B) on partial session resume, the merged restriction wasn't recomputed, leaving a stale later resetAt and stronger kind in the DB.
Confidence Score: 5/5Safe to merge. Both lifecycle gaps are closed, the synchronous-event ordering invariant is locked in by a dedicated test, and changes are additive with no regression risk to the existing rate-limit path. The banner-Cancel respawn relies on the session.rate_limit_resume event firing synchronously during stopSessionPreserveDb while subSessions still holds the session. The PR documents this thoroughly and a new test using a real InternalEventBus verifies the invariant. The recomputeTaskRestriction refactor applies the same merge semantics symmetrically on pause and resume, with all prior thread concerns addressed. Files Needing Attention: No files require special attention. The most complex logic in task-agent-manager.ts is well-commented and thoroughly tested.
|
| Filename | Overview |
|---|---|
| packages/daemon/src/lib/agent/rate-limit-watchdog.ts | Adds the narrow bannerCancelled boolean flag and exposes it via isRateLimitBannerCancelled(). Correctly scoped to distinguish banner-Cancel from a mid-fireCooldownRetry pause. |
| packages/daemon/src/lib/agent/agent-session.ts | Thin delegation: adds isRateLimitBannerCancelled() forwarding to the watchdog. |
| packages/daemon/src/lib/space/runtime/task-agent-manager.ts | Core of both fixes: upgrades limitedSessionsByTask, replaces markTaskRateLimited with synchronous recomputeTaskRestriction, adds respawnRateLimitedExecution, and changes resumeRateLimitedSubSession return type to retried |
| packages/daemon/src/lib/space/runtime/space-runtime.ts | Captures the new tri-state return and skips prepareSubSessionForWorkflowResume when outcome is respawned. |
| packages/daemon/tests/unit/1-core/agent/rate-limit-watchdog.test.ts | Adds two tests covering banner-only signal semantics and episode-boundary clearing. |
| packages/daemon/tests/unit/5-space/agent/task-agent-manager-cancel.test.ts | Full test suite for the respawn path including the synchronous-event stale-entry cleanup invariant using a real InternalEventBus. |
| packages/daemon/tests/unit/5-space/runtime/task-agent-rate-limit-listener.test.ts | Adds the multi-session partial-resume regression test verifying restriction shrinks to remaining sessions on partial resume. |
Reviews (4): Last reviewed commit: "fix(space): rate-limit task-lifecycle re..." | Re-trigger Greptile
lsm
left a comment
There was a problem hiding this comment.
🤖 Review by glm-5.1 (NeoKai)
Model: glm-5.1 | Client: NeoKai | Provider: Zhipu (GLM)
Recommendation: REQUEST_CHANGES — one P2 (respawn signal is too broad); the two core fixes themselves are correct.
Both core fixes are correct and I verified them end-to-end. 61/61 changed tests pass; lint + typecheck clean.
Finding A (banner-Cancel respawn) — correct. After respawnRateLimitedExecution resets the NodeExecution to pending + agentSessionId: null, the workflow tick selects it (space-runtime.ts:8041-8043 filters on status === 'pending' only) and spawns a fresh session via spawnWorkflowNodeAgentForExecution. recoverWorkflowBackedTask registers the executor (ensureExecutorRegistered) and the tick loop is always running, so the replacement IS spawned. Clearing startedAt/completedAt is actually more correct than the recoverRateLimitedTasks path it mirrors (that one leaves a stale startedAt that can trip the timeout detector around space-runtime.ts:7699).
Finding B (partial-resume recompute) — correct. Merge rules (max resetAt, strongest kind), the no-op skip, the terminal-status guard, and the empty → restoreTaskFromRateLimit path all check out, and the new regression test covers the documented max-resumes-first case.
P2 — the respawn signal also matches the in-flight auto-retry window
The respawn branch (task-agent-manager.ts:2339) fires when retryNowAfterRateLimit() is false and isRateLimitPauseActive() (= watchdog paused) is true. The watchdog doc correctly excludes an armed cooldown (there retryNow() returns true). But there's a second non-banner state where both hold: after the cooldown timer fires (cooldownTimer = null, rate-limit-watchdog.ts:427) while fireCooldownRetry is awaiting the retry callback (rate-limit-watchdog.ts:465). In that window paused is still true (notifyResume hasn't fired yet) and retryNow() returns false (no timer, not startup-exhausted) — so a manual Resume respawns a session whose auto-retry is actively starting, instead of letting it finish.
It's recoverable: cancel() bumps the generation, so the in-flight fireCooldownRetry self-aborts at its generation guard (rate-limit-watchdog.ts:472) and the turn re-runs via the fresh spawn. But it's unintended (only banner-Cancel should respawn), discards the in-flight retry, mints a throwaway session, and on a repeatedly-rate-limited task with an impatient user can repeat each cycle.
Suggested fix: gate the respawn on a signal set only by the banner path — e.g. a bannerCancelled flag set in cancel(notifyResume = false) and cleared on the next scheduleRetry / notifyPause / reset — rather than on paused. (Alternatively, also require that no episode is active / no retryCallback is in flight.)
P3 — change-skip guard doesn't compare the reason string
recomputeTaskRestriction's no-op guard (task-agent-manager.ts:790-797) compares status, resetAt, and type but not restrictions.limit (the reason). If two limited sessions share the same resetAt but differ in reason and one resumes (or a same-resetAt session pauses with a new reason), only the reason changes → the write is skipped → a stale reason is persisted. Observability/diagnostics only; no recovery impact. One-line fix: add task.restrictions?.limit === reason.
Note on the Greptile "stale limitedSessionsByTask entry" comment
That concern is unfounded. respawnRateLimitedExecution → stopSessionPreserveDb → handleInterrupt → watchdog.cancel(notifyResume = true) publishes session.rate_limit_resume, and InternalEventBus.publish runs the (synchronous) resume listener before its first await — i.e. synchronously inside the stop, before the eviction (agentSessionIndex.delete / subSessions cleanup). The listener's findParentTaskIdForSubSession still resolves the task (the session is still in subSessions) and deletes the entry, so the map is cleaned. (The respawn unit test doesn't exercise this because its fake session doesn't wire the bus — a follow-up test asserting the entry is gone after respawn would lock this in, but it's not a defect.)
…ful recompute skip guard Addresses greptile review on #2318. P1 — banner-Cancel respawn left a stale limitedSessionsByTask entry. The pause listener added the session's entry on the original cooldown; banner-Cancel (cancel(false)) never fires the resume event that would remove it, so it survived respawnRateLimitedExecution's eviction of agentSessionIndex/subSessions. When the replacement session later rate-limited and then resumed, the resume listener deleted only its own entry, saw the stale one still present, and recomputed instead of restoring — leaving the task stuck rate-limited until a daemon restart. respawnRateLimitedExecution now removes the abandoned session's entry and either recomputes (sibling still limited) or clears the now-stale restriction blob (last limited session gone — recoverWorkflowBackedTask had moved the task to in_progress via its transaction, which doesn't clear restrictions on its own). P2 — recomputeTaskRestriction's change-skip guard didn't compare the reason string (restrictions.limit), so a re-pause that only flipped the reason was silently skipped, persisting a stale reason. Added limit to the guard. Adds two regression tests: stale entry + restriction cleared on respawn, and recompute-from-sibling when another session is still limited.
…undant entry cleanup Addresses reviewer REQUEST_CHANGES on #2318 (round 2). P2 — the respawn branch was gated on the raw `paused` flag, but `paused` is also true (with retryNow() === false) during the brief window after the cooldown timer fires while fireCooldownRetry is mid-await — i.e. an auto-retry is actively starting. A manual Resume in that window would respawn a session whose retry is in flight (wasteful, discards the in-flight retry, and repeats each cycle on a repeatedly rate-limited task). Replaced it with a narrow `bannerCancelled` flag set ONLY in cancel(notifyResume=false) and cleared on the next pause/resume/new episode/reset. The respawn now fires only for genuinely banner-cancelled (parked) sessions; a Resume during an in-flight auto-retry is a clean no-op. Also removed the explicit limitedSessionsByTask cleanup + helper added in 1a59b4b — it was redundant. The entry is cleared by the session.rate_limit_resume event published synchronously inside stopSessionPreserveDb → handleInterrupt → cancel(notifyResume=true) → notifyResume → InternalEventBus.publish, which runs the resume listener (findParentTaskIdForSubSession + map delete) BEFORE the eviction, while subSessions still holds the session. (Verified: the bus delivers synchronously and notifyResume uses publish, not publishAsync.) Tests: isRateLimitBannerCancelled state-machine (idle / armed cooldown is NOT a banner / cancel(false) sets it / cancel(true) clears it / new episode clears it); respawn tests renamed to the banner signal; added a no-respawn-during-in-flight- retry case and a resume-event-clears-entry case (realistic handleInterrupt that publishes the resume event, locking in the invariant).
…n + partial-resume recompute Two P2 task-lifecycle edge cases deferred from #2271 (codex round-4 review), revised through two rounds of review. Finding A — banner Cancel left the task unrecoverable in-process. After cancelRateLimitRetry (cancel(false) + setIdle), retryNow could not re-fire (timer dropped, startupExhausted cleared) and the cross-restart recoverRateLimitedTasks sweep skipped the live idle session, so the visible Resume could not restart the consumed turn until a daemon restart. resumeRateLimitedSubSession now detects the parked state via a narrow `bannerCancelled` flag (set only in cancel(notifyResume=false), cleared on the next pause/resume/new-episode/reset — NOT the raw `paused` flag, which is also true mid-fireCooldownRetry while an auto-retry is actively starting) and re-spawns the execution (reset to pending + clear agentSessionId, stop + evict the orphaned idle session) so the workflow tick spawns a fresh replacement. recoverWorkflowBackedTask skips the now-redundant prepare step on respawn. Finding B — merged restriction was not recomputed on partial session resume. limitedSessionsByTask was IDs-only and the resume listener left the merged restriction untouched when other sessions remained, so the latest-deadline session resuming first left a stale later resetAt (delaying cross-restart recovery) and the status stayed at the stronger kind. Changed to Map<taskId, Map<sessionId, {resetAt,kind,reason}>> and recompute the merged restriction from all/remaining entries on every pause and resume, keeping the persisted restriction consistent with the in-memory set. The change-skip guard compares status/resetAt/type/limit so a re-pause that only flips the reason persists. The banner-cancelled session's limitedSessionsByTask entry is cleared by the session.rate_limit_resume event published synchronously inside stopSessionPreserveDb → handleInterrupt → cancel(notifyResume=true) → notifyResume → InternalEventBus.publish (verified: the bus delivers synchronously, notifyResume uses publish), which runs the resume listener before eviction while subSessions still holds the session — no explicit cleanup needed. Tests: multi-session partial-resume recompute; isRateLimitBannerCancelled state-machine (idle / armed cooldown is not a banner / cancel(false) sets it / cancel(true) + new episode clear it); respawn on Resume (respawned / retried / noop / noop-missing / no-respawn-during-in-flight-retry); resume-event-clears- entry (realistic handleInterrupt publishing the resume event).
468f550 to
fdaae26
Compare
lsm
left a comment
There was a problem hiding this comment.
🤖 Review by glm-5.1 (NeoKai)
Model: glm-5.1 | Client: NeoKai | Provider: Zhipu (GLM)
Recommendation: APPROVE
Round-2 verification, fresh after the rebase to fdaae2650. Both prior findings are resolved and correctly scoped; 64/64 rate-limit/respawn tests pass, typecheck + lint clean, PR is OPEN + MERGEABLE with all 4 review threads resolved (none outdated). No outstanding findings.
P2 resolved — banner-only respawn gate. The respawn now keys off a new bannerCancelled flag instead of the raw paused flag. I confirmed bannerCancelled is set in exactly one place — the else branch of cancel() (notifyResume=false) in rate-limit-watchdog.ts — and that path is reached only from the cooldown banner's Cancel button (RateLimitCooldownBanner → session.cancelRateLimitRetry RPC → cancelRateLimitRetry() → cancel(false)); there are no other cancel(false) callers. It is cleared on every transition that should retire it (new episode in scheduleRetry, reset, notifyPause, notifyResume). During the in-flight fireCooldownRetry window the flag stays false, so a manual Resume there is a clean no-op rather than a wasteful respawn. New tests lock both halves in (banner-only-signal, new-episode-clears, and "does NOT respawn while an auto-retry is in flight").
P3 resolved — reason in skip-guard. restrictions.limit is now compared in the no-op guard, so a re-pause that only flips the reason persists.
Rebase note. recomputeTaskRestriction now uses isRateOrUsageLimited(task.status) from #2317; that helper is behavior-identical to the old inline rate_limited || usage_limited check, just consolidated.
Entry-cleanup (the Greptile concern). respawnRateLimitedExecution now documents why no explicit limitedSessionsByTask cleanup is needed (the synchronous session.rate_limit_resume fired inside handleInterrupt runs the resume listener before eviction and deletes the entry), and a new test asserts the map is empty after respawn — so a replacement session's later pause/resume resolves cleanly.
Fixes two P2 rate-limit task-lifecycle edge cases deferred from #2271.
Finding A — banner Cancel left the task unrecoverable in-process. The cooldown banner's Cancel (
cancel(false)+setIdle) left the task rate-limited butretryNowcouldn't re-fire and the cross-restart sweep skipped the live idle session, so the visible Resume couldn't restart the consumed turn until a daemon restart.resumeRateLimitedSubSessionnow detects the parked state via a narrowbannerCancelledflag (set only bycancel(notifyResume=false)— narrower than the rawpausedflag, which is also true mid-fireCooldownRetrywhile an auto-retry is actively starting) and re-spawns the execution (reset topending+ clearagentSessionId, stop + evict the idle session) so the workflow tick spawns a fresh replacement. The stalelimitedSessionsByTaskentry is cleared by the synchronoussession.rate_limit_resumeevent fired insidehandleInterrupt.Finding B — merged restriction wasn't recomputed on partial session resume. The resume listener left the merged restriction untouched when other sessions remained, so the latest-deadline session resuming first left a stale later
resetAt(delaying cross-restart recovery).limitedSessionsByTaskis nowMap<taskId, Map<sessionId, {resetAt,kind,reason}>>and recomputes (later resetAt + stronger kind + reason) from all/remaining entries on every pause and resume, keeping the persisted restriction consistent with the in-memory set.Both changes are strictly safer than the prior behavior.