Backfill content for docs catalogued while their source fetch failed

[qiuethan]

A doc catalogued while its source fetch fails keeps no content, and nothing can ever fill it in except a manual per-doc refetch.

The gap

ingest_doc is deliberately resilient: a FetchError becomes a warning and the doc is still catalogued, so a connectors outage or a missing credential never fails POST /docs. That is the right behaviour — but it leaves the doc with no doc_content row, and there is no path back.

Re-ingesting the same URL does not help. The dedup check (src/ingest.py, step 1) matches the existing active doc and returns via _merge_into_existing before the fetch step runs. It folds in new tags and grants and returns created=False. It never fetches. So the obvious operator instinct — "just add it again" — is a no-op for content.

The only repair is POST /docs/{id}/refetch, one document at a time, and there is no way to enumerate which documents need it short of querying for a null doc_content directly against the database.

Why this is about to matter

Migration 006 (#84) flips content_fetch_enabled to true for four Google sources at once. Before it, the auth-gated sources never attempted a fetch, so a content-less Google doc was normal and expected. After it, a content-less Google doc means something went wrong — but it looks identical to a doc that was catalogued before the migration.

So the first connectors outage, the first expired credential, or the first period before the Google service account is provisioned produces a cohort of content-less docs that is invisible to the API. The helper bot and RAG (#125) will silently ground on a partially-populated corpus.

The same gap applies to every doc catalogued before #83 added doc_content at all.

Scope

• A way to find them. Either a filter on the docs list (has_content=false) or a CLI query. This is the part that is missing entirely — right now the state is only visible in SQL.
• A way to fix them in bulk. A backfill-content command that iterates docs lacking a doc_content row, refetches, and upserts. services/documentation-system/src/cli.py exists but is doc-keys, explicitly and only API-key management — a prog="doc-keys" argparse with three key subcommands. A backfill-content subcommand there would be a poor fit; a second console script (doc-backfill) alongside it matches the repo's one-script-per-purpose shape better. Decide in the plan.
• Decide whether re-ingest should backfill. Making _merge_into_existing attempt a fetch when the doc has no content would close the loop for the common case, at the cost of turning a cheap idempotent dedup into a network call. Probably not — but it should be a decision rather than an omission, and the current behaviour deserves a comment either way.
• Respect visibility: any listing must go through the same actor-scoped path as every other read (doc_visible), or be explicitly admin-only.

Tests

• A doc ingested while the fetcher raises has no doc_content row and carries a warning.
• Re-ingesting that URL still does not populate content (pins the current dedup behaviour, whichever way the decision above goes).
• The backfill command finds exactly the content-less docs and populates them.
• A doc whose source still fails stays content-less and is reported, not silently skipped.

Related: #83 (added doc_content), #84 (enabled fetching for the Google sources), #125 (RAG grounds on this content).