Skip to content

fix: consolidation batch failure must not report a succeeded job - #33

Merged
chrlesur merged 2 commits into
mainfrom
fix/consolidation-batch-failure-status
Jul 8, 2026
Merged

fix: consolidation batch failure must not report a succeeded job#33
chrlesur merged 2 commits into
mainfrom
fix/consolidation-batch-failure-status

Conversation

@chrlesur

@chrlesur chrlesur commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Closes #32

Context

Production incident (2026-07-08, space terraform-provider): the LLMaaS backend rejected every consolidation call (context-length error), yet bank_consolidate jobs ended succeeded with notes_processed=0 and batches=0/50. The failure stayed invisible for 6 days. The root cause of the LLM rejection itself was an infrastructure misconfiguration (handled separately on the LLMaaS side); this PR fixes the Live Memory reporting bug that masked it.

Changes

consolidator.pyconsolidate() now tracks the first batch failure (LLM or bank write) instead of silently breaking out of the loop:

  • ok — no failure (unchanged contract);
  • error — failure with 0 completed batches (nothing was integrated);
  • partial — failure after ≥1 applied batch (work was integrated and the corresponding notes deleted; metrics preserved);
  • failed runs expose failed_batch and a message embedding the upstream error plus the number of notes left in live/.

consolidation_queue.py — the worker already mapped any non-ok status to a failed job with result.message as error; the lane phase now reflects the real outcome (done/failed) instead of always displaying done.

config.py — fail fast at startup when LLMAAS_MAX_TOKENS >= LLMAAS_CONTEXT_WINDOW: such a budget can never produce a valid call on a large prompt.

Tests

  • tests/test_consolidation_batch_failure.py (7 tests) replays the incident: first batch rejected with the real 262144 context-length error → asserts status="error", zero metrics, _meta.json untouched, no bank write attempted, and end-to-end through the queue a failed job with an exploitable error. Partial-failure and full-success non-regression paths covered.
  • tests/test_config.py::TestLlmBudgetCoherence (4 tests) for the startup guard.
  • Adversarial check: with the fix reverted (git stash on src/), 6 of the new tests fail on the original code — they bite, they don't rubber-stamp.
  • Full suite: 424 passed + 1 xfailed (baseline was 413 + 1).

Validation plan (post-merge, production)

  1. Redeploy, then re-run bank_consolidate(space_id="terraform-provider") (107 pending notes).
  2. Expect notes_processed > 0, llm_tokens_used > 0, live.notes_count decreasing, last_consolidation updated.
  3. Counter-check: a forced LLM failure must now end the job failed with the upstream message — never a silent empty succeeded.

chrlesur added 2 commits July 8, 2026 17:18
A failed batch (LLM or bank write) used to break out of the loop while
the final result was still built with a hardcoded "status": "ok". The
queue mapped it to a succeeded job even with 0/50 batches completed,
which kept the terraform-provider incident invisible for 6 days.

- track the first batch failure (message + 1-based index)
- status is now: ok (no failure), error (failure with 0 completed
  batches), partial (failure after at least one applied batch)
- failed runs expose failed_batch and an actionable message including
  the upstream error and the number of notes left in live/
- queue lane phase reflects the outcome (done/failed) instead of
  always displaying done

Adversarial tests replay the incident (first batch rejected with the
real 262144 context-length error) and assert status, metrics,
_meta.json untouched and no bank write attempted.

Refs #32
…rtup

An output budget that consumes the whole context window can never
produce a valid call: the backend rejects input + max_tokens > window
before inference. Fail fast at startup instead of failing every
consolidation at runtime.

Refs #32
@chrlesur
chrlesur merged commit b9c6647 into main Jul 8, 2026
2 checks passed
@chrlesur chrlesur mentioned this pull request Jul 8, 2026
pull Bot pushed a commit to moul/live-memory that referenced this pull request Jul 8, 2026
Bump VERSION, __version__ and version assert to 2.5.4; add CHANGELOG
entry for the batch-failure status fix (Cloud-Temple#32/PR Cloud-Temple#33), the /live stale
selectBank guard (Cloud-Temple#24/PR Cloud-Temple#28), and the LLM budget startup guard.
Refresh the stale version badge (2.5.1 → 2.5.4) in both READMEs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bank_consolidate reports succeeded when the first LLM batch fails (0 notes processed)

1 participant