fix: consolidation batch failure must not report a succeeded job - #33
Merged
Conversation
A failed batch (LLM or bank write) used to break out of the loop while the final result was still built with a hardcoded "status": "ok". The queue mapped it to a succeeded job even with 0/50 batches completed, which kept the terraform-provider incident invisible for 6 days. - track the first batch failure (message + 1-based index) - status is now: ok (no failure), error (failure with 0 completed batches), partial (failure after at least one applied batch) - failed runs expose failed_batch and an actionable message including the upstream error and the number of notes left in live/ - queue lane phase reflects the outcome (done/failed) instead of always displaying done Adversarial tests replay the incident (first batch rejected with the real 262144 context-length error) and assert status, metrics, _meta.json untouched and no bank write attempted. Refs #32
…rtup An output budget that consumes the whole context window can never produce a valid call: the backend rejects input + max_tokens > window before inference. Fail fast at startup instead of failing every consolidation at runtime. Refs #32
Merged
pull Bot
pushed a commit
to moul/live-memory
that referenced
this pull request
Jul 8, 2026
Bump VERSION, __version__ and version assert to 2.5.4; add CHANGELOG entry for the batch-failure status fix (Cloud-Temple#32/PR Cloud-Temple#33), the /live stale selectBank guard (Cloud-Temple#24/PR Cloud-Temple#28), and the LLM budget startup guard. Refresh the stale version badge (2.5.1 → 2.5.4) in both READMEs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #32
Context
Production incident (2026-07-08, space
terraform-provider): the LLMaaS backend rejected every consolidation call (context-length error), yetbank_consolidatejobs endedsucceededwithnotes_processed=0andbatches=0/50. The failure stayed invisible for 6 days. The root cause of the LLM rejection itself was an infrastructure misconfiguration (handled separately on the LLMaaS side); this PR fixes the Live Memory reporting bug that masked it.Changes
consolidator.py—consolidate()now tracks the first batch failure (LLM or bank write) instead of silently breaking out of the loop:ok— no failure (unchanged contract);error— failure with 0 completed batches (nothing was integrated);partial— failure after ≥1 applied batch (work was integrated and the corresponding notes deleted; metrics preserved);failed_batchand amessageembedding the upstream error plus the number of notes left inlive/.consolidation_queue.py— the worker already mapped any non-okstatus to afailedjob withresult.messageaserror; the lanephasenow reflects the real outcome (done/failed) instead of always displayingdone.config.py— fail fast at startup whenLLMAAS_MAX_TOKENS >= LLMAAS_CONTEXT_WINDOW: such a budget can never produce a valid call on a large prompt.Tests
tests/test_consolidation_batch_failure.py(7 tests) replays the incident: first batch rejected with the real 262144 context-length error → assertsstatus="error", zero metrics,_meta.jsonuntouched, no bank write attempted, and end-to-end through the queue afailedjob with an exploitableerror. Partial-failure and full-success non-regression paths covered.tests/test_config.py::TestLlmBudgetCoherence(4 tests) for the startup guard.git stashonsrc/), 6 of the new tests fail on the original code — they bite, they don't rubber-stamp.Validation plan (post-merge, production)
bank_consolidate(space_id="terraform-provider")(107 pending notes).notes_processed > 0,llm_tokens_used > 0,live.notes_countdecreasing,last_consolidationupdated.failedwith the upstream message — never a silent emptysucceeded.