Epic: RAG retrieval layer — /embed + pgvector store + actor-scoped similarity search

[qiuethan]

Blocked by

No response

Summary

The helper-bot epic (#81) is meant to answer questions grounded in our docs via RAG. Content acquisition (#82) covers getting the text; nothing else on the board covers turning that text into retrievable vectors and querying them. This epic is the retrieval layer that sits between #82 and #81.

This body replaces the original scope. Four things were wrong with it:

1. It delivered nothing anyone could use. All six sub-issues were plumbing, so the epic could complete in full and no member would notice — value only appeared once Epic: Member-facing helper bot (LLM-backed, docs-grounded Q&A) #81 wired it up. A /doc search surface is now the last step, so the epic lands something members use and Epic: Member-facing helper bot (LLM-backed, docs-grounded Q&A) #81 becomes "make it conversational" rather than "make it exist."
2. Nothing triggered indexing. "Re-index on change" had no caller: this platform has no scheduler, no queue, and no background worker anywhere, and docs/ARCHITECTURE.md rules out running one inside a source of truth. Resolved in the Model below.
3. Model changes had no owner. RAG: pgvector store + embeddings schema (documentation-system) #174 records the embedding model per row "so a model change is detectable", but no issue said what happens when it is detected. Re-embedding is both a migration and a Bedrock cost event.
4. Two blocking design questions sat in prose. Chunk size/overlap and hybrid-vs-pure-vector are decisions the pipeline gets built on top of, so they move to a spike that runs first and produces an answer.

#176 and #177 are merged. Visibility filtering has to happen inside the retrieval query rather than after ranking, so "build the query" and "filter the query" are not separable pieces of work — split either way round, one of them is unstartable. They become one issue whose acceptance includes the authorization tests, and #177 is closed as superseded.

Model

Four stages, each independently testable: decide (spike the chunking and search strategy against real catalogued content), embed (LLM service gains /embed), index (chunk + upsert into pgvector on documentation-system), retrieve (top-k similarity search, actor-scoped and citable, in one endpoint) — then a thin surface so the capability is visible to members.

Two constraints govern the shape:

• Retrieval is an authorization boundary. The LLM is not a trust boundary. A user must never retrieve a chunk from a document they cannot read, regardless of how the question is phrased. Filtering happens in the query, not after ranking — otherwise top-k is computed over documents the user cannot see and returns fewer results than it should. This is why retrieval and visibility are a single issue rather than two.
• Nothing runs inside a source of truth. Indexing is triggered by the ingest path that already exists in src/ingest.py — which is already fetching content at that point — plus an explicit authenticated admin re-index endpoint for backfill and repair. No job runner lives inside the catalog, so this stays consistent with the API-only principle rather than becoming an exception to it.

Index catalog entities, not URL-documents. The pipeline keys on whatever the catalog holds, not on the URL-fetch path that happens to populate it today. Meeting records are coming (see the meeting-knowledge epic) and will be catalogued entities with no source URL. A pipeline written around URL-fetched docs silently excludes them, and that gets discovered months later when the helper bot cannot answer a question about a decision the org definitely made.

Corpus reality: only the Google connector is built — services/connectors/src/sources/ holds google.py and nothing else. Notion (#85) and GitHub (#86) widen the corpus but are not required to start.

Decomposition (build order)

• Spike: chunking + search strategy — chunk size/overlap for the doc shapes we actually have, and whether hybrid (keyword + vector) beats pure vector at our corpus size. The decision lands in services/documentation-system/docs/ARCHITECTURE.md and in RAG: chunking + indexing pipeline (embed, upsert, re-index, delete) #175's design notes, not in a comment thread.
• RAG: /embed endpoint on the LLM service (Bedrock-backed) #173 — Embedding provider — /embed on the LLM service, Bedrock-backed, mirroring the existing provider abstraction and API-key auth. Fixes the embedding model, and with it the vector dimensionality RAG: pgvector store + embeddings schema (documentation-system) #174 needs.
• RAG: pgvector store + embeddings schema (documentation-system) #174 — Vector store — pgvector extension on the documentation-system database; embeddings table keyed to document + chunk, recording model and dimensionality per row. Blocked by RAG: /embed endpoint on the LLM service (Bedrock-backed) #173: vector(N) takes its dimension at column definition and both index types want it fixed, so the schema cannot be written before the model is chosen.
• RAG: chunking + indexing pipeline (embed, upsert, re-index, delete) #175 — Chunking + indexing pipeline — chunk extracted text, embed, upsert; idempotent re-index; delete removes chunks. Keyed on catalog entities rather than the URL-fetch path. Triggered inline on ingest, plus an authenticated admin re-index endpoint.
• NEW — Re-embed on model change — detect a dimensionality/model mismatch, re-embed affected chunks, and make the cost and duration of a full re-index a known quantity rather than a surprise.
• RAG: retrieval endpoint — actor-scoped top-k similarity search with source citations #176 — Retrieval endpoint, actor-scoped — top-k similarity search returning citable source references, filtered in-query by the per-doc grants from Document visibility: per-doc grants (person/team) + actor-scoped enforcement #74. Configurable k and a relevance floor, so a query with no good match returns nothing instead of the least-bad chunk. Absorbs RAG: actor-scoped visibility enforcement on retrieval #177.
• RAG: retrieval quality evaluation set #178 — Retrieval quality evaluation set — a small hand-built question set with expected sources, and a harness reporting hit-rate at k so chunking changes are measured rather than guessed at.
• NEW — /doc search — a member-facing search subcommand over retrieval, returning cited docs. The slice that makes this epic visible.

Area

service, docs-system, bot

What we can build on

• Content acquisition: extend FetchResult + storage for full extracted text (precursor) #83 landed full extracted text in storage and Google Workspace content connector (Docs/Drive/Sheets/Slides) #84 landed the Google Workspace connector, so there is real content to index today.
• Document visibility: per-doc grants (person/team) + actor-scoped enforcement #74 built per-doc grants and actor-scoped enforcement — visibility filtering reuses doc_visible() / doc_grants rather than reinventing them.
• The LLM service already has a provider abstraction and API-key auth (LLM service: provider abstraction + Claude implementation #42, LLM service: API-key auth for internal callers #41), so /embed extends an established pattern rather than adding new surface.
• documentation-system already fetches content during ingest (src/ingest.py), which is the hook the indexing trigger uses.
• documentation-system-test already runs migrations against real Postgres in CI, so the pgvector migration is covered by an existing job.

Not in this epic

• Content acquisition — fetching and extracting source text is Epic: Content acquisition — authenticated source connectors (documentation-system) #82's job. This epic consumes what that produces.
• The bot's conversational surface — Epic: Member-facing helper bot (LLM-backed, docs-grounded Q&A) #81 owns LLM-backed Q&A. This epic ships search, not a chat experience.
• Tool-use / function-calling — LLM service: tool-use / function-calling support in /chat #70 and Helper bot: agentic tool-calling — expose llmSafe services as tools #71 are a separate capability line; retrieval informs them but does not depend on them.
• Making meeting records exist — the meeting-knowledge epic owns that. This epic only has to not exclude them, which is what the catalog-entity requirement above is for.
• A general scheduled-jobs capability — deliberately avoided. Docs visibility: auto-sync grants from source ACLs (Drive/Figma → people) #76, Backfill content for docs catalogued while their source fetch failed #159, and Observability — Tier 2: external uptime + Discord alerting #55 all want one; if it lands later, the admin re-index endpoint becomes one of its callers rather than being replaced.

Open questions

• Should the relevance floor be absolute or relative to the top hit? An absolute threshold behaves badly across differently-sized corpora.
• Does /doc search respect the same visibility rules as /doc list, or is search deliberately narrower to start?