fix: report the typed-fact backlog nothing will drain - #2322
Merged
Conversation
Two gates that do not agree.
memory.store enqueues a "memory_facts" job whenever typed_facts_enabled is
on, which is the DEFAULT. The only consumer of those jobs is
kb_memory_facts_drain, which runs on the curator LLM lane -- and that lane
deliberately does not start without a synthesis endpoint ("NO SYNTHESIS
PROVIDER => NO LLM LANE"). Both decisions are individually correct.
Together they mean an install with no synthesis provider, which is a
supported configuration, enqueues one row per stored memory that nothing
will ever claim.
Found by exploratory testing of :testing, not by a failing check. On the
e2e VM: 4 memory_facts jobs pending for 11.5 hours with attempts=0, while
/v1/health reported status ok with an empty warnings array and `aimee kb
status` printed "4 pending" with nothing to say that pending here means
forever. It grows without bound for the life of the install.
Reported as a WARNING, not a blocker. Running without a synthesis provider
is explicitly supported and memory store and search work perfectly; only
typed-fact enrichment is deferred. Degrading the verdict for a supported
configuration is exactly the dilution the status derivation exists to
avoid. The message says the backlog is not lost: configuring a provider
drains it.
Counting needs PENDING specifically, so add
db2_kb_async_count_kind_pending alongside the existing total. Total
conflates work still waiting with work long finished and cannot tell a
queue that is draining from one that never will -- which is the entire
distinction being reported. Test red-verified by dropping the status
filter.
The kb target compiles kb_service_kb.c with a different include set than the server target, so config_synth_chat_endpoint_current was an implicit declaration there and `make ../aimee-kb` failed under -Werror. The container build is that target, so the image build broke while the default target still passed.
The backlog warning added in the previous commit reached /v1/health and stopped there. server_health_add_kb passed blockers through and dropped the kb's warnings array on the floor, and the status renderer only ever printed blockers. Verified live before this commit: the kb reported the 4-job typed-fact backlog in its warnings array, while `aimee status` printed "aimee-kb: ok" with four clean detail lines and `aimee kb status` printed a bare "4 pending". Publishing a finding into a field no surface renders is the same defect as never computing it -- the evidence exists and the summary does not carry it, which is the whole shape this series is about. Warnings are advisory by construction: they do not move the verdict, and that is exactly why they need printing. A kb that is genuinely ok can still be accumulating work nothing will process, and a status line that renders only blockers reports that as a clean bill of health. Rendered as "note:" to keep it visibly distinct from "BLOCKED:", which means cannot work. Extracted kb_health_add_lines: two copies of the same newline-splitting arithmetic is how the warnings array came to be dropped to begin with. Test asserts warnings survive with the verdict unchanged, and that the blockers key is absent rather than empty when there are none. Red verified by removing the pass-through.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two gates that do not agree
memory.storeenqueues amemory_factsjob whenevertyped_facts_enabledis on — the default:The only consumer of those jobs is
kb_memory_facts_drain, which runs on the curator LLM lane. That lane deliberately does not start without a synthesis endpoint:Both decisions are individually correct. Together they mean an install with no synthesis provider — a supported configuration — enqueues one row per stored memory that nothing will ever claim.
Found by exploratory testing, not by a failing check
On the e2e VM:
memory_factsjobs pending for 11.5 hours,attempts=0— never even claimed/v1/health→status: ok,warnings: [],blockers: []aimee kb status→Queue: 4 pending, 0 running, 0 failed, with nothing to say that pending here means foreverIt grows without bound for the life of the install. Same family as #2319 and #2321: the evidence existed, the summary did not carry it.
Reported as a warning, not a blocker
Running without a synthesis provider is explicitly supported, and memory store and search work perfectly — only typed-fact enrichment is deferred. Degrading the verdict for a supported configuration is exactly the dilution the status derivation added in #2319 exists to avoid. This is the distinction that PR built, being used as intended.
The message also says the backlog is not lost: configuring a provider drains it. That is why the fix is not "stop enqueueing" — the queue is a backfill, and dropping it would silently forfeit facts for every memory stored before a provider was configured.
Counting needs
pending, not totaldb2_kb_async_count_kindcounts every row regardless of status. That cannot distinguish a queue that is draining from one that never will — which is the entire distinction being reported — and would keep warning about work that already completed. Addeddb2_kb_async_count_kind_pendingalongside it.Verification
Test asserts pending tracks the backlog rather than the row total: two enqueued → 2 pending / 2 total; one marked done → 1 pending / 2 total; an
extract_docrow in the shared table is not counted in. Red verified by dropping thestatus = 'pending'filter — fails on exactly the pending-vs-total assertion.Lockfile:
kb-synthesissource pin resynced, recomputed with the exporter's own digest functions.Gate: clang-format →
make lint(41/41) →check-docs.py→check_c_repository_lock.py— all green.