RAG: retrieval endpoint — actor-scoped top-k similarity search with source citations

[qiuethan]

Blocked by

#175

What

The retrieval endpoint for the RAG epic (#125): top-k similarity search over indexed chunks, returning citable source references, with results filtered to what the asking actor is allowed to read.

This issue absorbs #177, which is closed as superseded. Retrieval and visibility were originally split across two issues and that split does not survive contact with the requirement — see Design notes.

Why

Indexing content nobody can query is a store, not a retrieval layer. This is the endpoint the evaluation set (#178), the /doc search surface, and eventually the helper bot (#81) all consume.

Authorization is not a follow-up. The LLM is not a trust boundary, and neither is a Discord command — a member must never retrieve a chunk from a document they cannot read, no matter how the question is phrased. Shipping search first and filtering second would mean a window in which the fastest way to read someone else's document is to ask about it.

Area

service, docs-system

Done looks like

• A query returns ranked chunks with resolvable source references — id, title, and chunk location — so an answer can cite rather than assert.
• Results are filtered by the per-doc grants from Document visibility: per-doc grants (person/team) + actor-scoped enforcement #74, applied in the query, not after ranking.
• Two actors with different grants issue the same query and get different results; an actor with no grants gets an empty result.
• Tests cover filter-before-rank specifically, not just the final result set — a post-filter implementation must fail them.
• k is configurable, and a relevance floor means a query with no good match returns nothing rather than the least-bad chunk.
• No route is reachable without actor scoping — there is no unfiltered variant, not even behind a flag.

Design notes

Why this is one issue and not two. Filtering has to happen inside the similarity query. Rank-then-filter computes top-k across documents the actor cannot see and then discards some, so a user with partial access silently gets fewer results than they asked for, and the more restricted they are the emptier their results get. Filter-then-rank is the only correct order, which means the visibility predicate is part of the query rather than a wrapper around it. Split across two issues, whichever ships first is either wrong or unstartable.

Reuse doc_visible() / doc_grants from #74 rather than reimplementing the visibility rule. A second implementation of an authorization predicate is a second place for it to be wrong.

Whether the relevance floor is absolute or relative to the top hit is still open on the epic — an absolute threshold behaves badly across differently-sized corpora, and our corpus size is going to change a lot.

Out of scope

• The chunking and indexing that populate the store — RAG: chunking + indexing pipeline (embed, upsert, re-index, delete) #175.
• Measuring whether retrieval is any good — RAG: retrieval quality evaluation set #178 owns the evaluation set.
• Any member-facing surface — /doc search is its own issue, and conversational answers are Epic: Member-facing helper bot (LLM-backed, docs-grounded Q&A) #81.