refactor: knowledge-base schema (issue #98) — phases 1a–1d

[erichare]

Summary

Long-lived feature branch for #98. Each phase landed here as its own commit; ready to merge.

• ✅ 1a — schema, types, converters, bundle wiring
• ✅ 1b — knowledge-base + 3 service CRUD routes
• ✅ 1c.1 — drop saved-queries surface
• ✅ 1c.2 — KB data plane + auto-provisioning
• ✅ 1c.3 — drop legacy /catalogs and /vector-stores
• ✅ 1c.4 — wire web UI to KB endpoints
• ✅ 1d — switch documents/ingest/search to KB scope
• 2a — LLM + MCP service CRUD
• 2b — agent execution loop
• 2c — conversations + messages + UI

Phase 1a — schema (commit 31d8488)

11 new tables created additively — wb_config_*, wb_rag_*, wb_agentic_*. Row types, control-plane records (camelCase), and converters added. Pure addition, no behaviour change.

Layer	Tables
wb_config_*	workspaces, knowledge_bases_by_workspace, chunking/embedding/reranking/llm/mcp_tools
wb_rag_*	documents_by_knowledge_base, _by_status, _by_content_hash
wb_agentic_*	agents_by_workspace, conversations_by_agent (DESC), messages_by_conversation

Phase 1b — CRUD routes (commit 6e30717)

Four new route files under /api/v1/workspaces/{workspaceUid}/:

• /knowledge-bases — full CRUD; auto-provisions vector collection on create
• /chunking-services — CRUD
• /embedding-services — CRUD
• /reranking-services — CRUD

Twenty new ControlPlaneStore methods, implemented across memory, file, and astra backends.

Invariants enforced at the route layer:

• embeddingServiceId / chunkingServiceId immutable after KB create. Update schema is .strict() so PUT bodies including those keys → 400.
• Service deletion refused (409) when any KB still references it.
• vector_collection auto-named wb_vectors_<kb_id> (hyphen-stripped — Astra naming rules).

Set vs array trade-off: the schema uses SET<TEXT> for supported_languages / supported_content / tags. Astra row layer keeps Sets; the in-memory record exposes them as sorted, deduplicated readonly string[]. Converters normalise at the boundary.

Phase 1c.1 — drop saved-queries (commit 92a49ff)

The new schema does not include saved queries — the proposed agent layer (Stage 2) replaces the use case. Deleted route, store methods (5 each across 3 backends), DDL + bundle entry, OpenAPI schemas, and 19 tests.

Phase 1c.2 — KB data plane (commit 9ddff50)

Knowledge bases now own their underlying vector collection end-to-end:

1. resolveKb (kb-descriptor.ts) materialises a VectorStoreRecord-shaped descriptor on the fly from a KB + its bound embedding/reranking services. The driver and dispatch layers consume the descriptor unchanged.
2. KB CRUD provisions/drops the collection: POST …/knowledge-bases creates the row then driver.createCollection, with rollback on provisioning failure; DELETE drops the collection first, then the row.
3. New data-plane endpoints under …/knowledge-bases/{kb}: POST records (upsert), DELETE records/{id}, POST search (vector / hybrid / rerank).

Phase 1d — KB-scoped documents + ingest (commit e62ecbe)

New endpoints under …/knowledge-bases/{kb}/:

GET    /documents              list (paginated)
POST   /documents              register a doc
GET    /documents/{d}          fetch
PUT    /documents/{d}          patch metadata
DELETE /documents/{d}          drop doc + cascade chunks
GET    /documents/{d}/chunks   list by index
POST   /ingest                 sync chunk + embed + upsert
POST   /ingest?async=true      202 + job pointer

Control plane: five new ControlPlaneStore methods for RAG documents (list/get/create/update/delete) backed by wb_rag_documents_by_knowledge_base. Astra writes maintain the by-status secondary index; by-content-hash gets written on create + content-hash changes. deleteKnowledgeBase cascades RAG document rows.

Job layer: JobRecord gains knowledgeBaseUid; wb_jobs_by_workspace gets a knowledge_base_uid column. New runKbIngestJob async worker resolves the KB descriptor on each call so renames / service swaps can't drift mid-flight.

Pipeline: runKbIngest is the KB-scoped sibling of runIngest. New KB_SCOPE_KEY = \"knowledgeBaseUid\" payload key gets stamped on every chunk for KB-scoped filtering.

Phase 1c.3 — drop legacy /catalogs + /vector-stores (commit 031af39)

Retires the catalog/vector-store/document surface in favour of the KB-scoped equivalents.

Routes deleted: /catalogs, /catalogs/{c}/documents, /catalogs/{c}/documents/search, /catalogs/{c}/documents/{d}/chunks, /catalogs/{c}/ingest, /vector-stores, /vector-stores/{vs}/..., /vector-stores/discoverable, /vector-stores/adopt.

Control plane: 15 ControlPlaneStore methods removed (catalog × 5, vector-store × 5, document × 5). Memory/file/astra implementations stripped along with their assert helpers. Workspace delete now resolves each KB into a driver descriptor via resolveKb and drops the underlying collection — no longer walks wb_vector_store_by_workspace.

Schema: DDL dropped: wb_catalog_by_workspace, wb_vector_store_by_workspace, wb_documents_by_catalog. Astra row types and converters for them go too. wb_jobs_by_workspace.catalog_uid column dropped (idempotent CREATE TABLE doesn't drop columns, so deployed schemas still have the dead column — harmless, ignored on read/write). JobRecord.catalogUid removed.

OpenAPI: Catalog, VectorStore, Document, AdoptableCollection, AdoptCollectionInput, Ingest* (catalog-shaped) schemas retired. KB-shaped equivalents (KnowledgeBase, RagDocument, KbIngestRequest, KbAsyncIngestResponse) are the only document surface.

Pipeline + worker: runIngest (catalog-scoped) and runIngestJob removed. runKbIngest + runKbIngestJob are the only ingest path. The cross-replica orphan sweeper now resumes via runKbIngestJob. CATALOG_SCOPE_KEY removed; KB_SCOPE_KEY is the only chunk-payload scope key.

Tests: app.test.ts trimmed from 3129 → 1023 lines: ~2100 lines of legacy catalog / vector-store / document / ingest / chunk / adopt tests deleted. Replacement coverage lives in knowledge-bases.test.ts (15 KB-scoped route tests) and control-plane/contract.ts (RAG document × 3 backends). Converters test rewritten around RagDocumentRecord. Astra-fake bundle stripped of catalogs / vectorStores / documents tables.

Conformance: 10 legacy fixtures deleted (catalog-*, vector-store-*, document-crud-basic). 5 workspace-only scenarios remain. KB scenarios deferred to a follow-up once the cross-runtime KB surface stabilises.

The legacy wb_workspaces and wb_jobs_by_workspace tables stay in place — both are used unchanged by the KB-scoped surface. Migrating to wb_config_workspaces is a separate refactor.

Phase 1c.4 — web UI rewire (commit 8534c9a)

The React UI now speaks the new KB-scoped surface end-to-end.

Schemas + API (apps/web/src/lib/schemas.ts, api.ts): trimmed of every catalog/vector-store/document/saved-query type and helper. Replaced with KB-shaped schemas (KnowledgeBaseRecord, chunking/embedding/reranking service records, RagDocumentRecord) and matching create/update/delete API calls. JobRecord.catalogUid → knowledgeBaseUid.

Hooks:

• Deleted: useCatalogs, useVectorStores, useSavedQueries
• Added: useKnowledgeBases, useServices (factory-style for chunking/embedding/reranking)
• Updated: useDocuments, useIngest, usePlaygroundSearch — all KB-scoped

Pages:

• WorkspaceDetailPage swaps VectorStoresPanel + CatalogsPanel for ServicesPanel + KnowledgeBasesPanel
• CatalogExplorerPage → KnowledgeBaseExplorerPage. Route changes from /workspaces/:wid/catalogs/:cid → /workspaces/:wid/knowledge-bases/:kbid
• PlaygroundPage picks workspace + KB instead of workspace + vector store. The query form takes a QueryFormTarget (vectorDimension + provider description + lexical/rerank flags) so it doesn't need to know about descriptors
• OnboardingPage copy updated

Components:

• Deleted: CatalogsPanel, VectorStoresPanel, AdoptCollectionDialog, CreateCatalogDialog, CreateVectorStoreDialog, SavedQueriesSection
• Added: KnowledgeBasesPanel, ServicesPanel, CreateKnowledgeBaseDialog
• Updated: DocumentTable, DocumentDetailDialog, IngestQueueDialog — switched to RagDocumentRecord and KB-scoped props (knowledgeBase / knowledgeBaseUid, documentId instead of documentUid, contentHash instead of md5Hash)

Tests: QueryForm.test, DocumentTable.test, IngestQueueDialog.test rewritten for the new shapes. golden-path.spec.ts (Playwright) walks the new flow: onboard → service creation (via API) → KB → upsert → playground.

CI fixes (commits 36d4d71, 2377bc8)

Two issues surfaced after the initial push:

1. Biome formatting + organise-imports drift across the touched TS/TSX files. biome check --write was idempotent on the runtime side; the one manual fix was let resolved in runKbIngestJob — Biome forbids implicit-any declarations, now annotated as Awaited<ReturnType<typeof resolveKb>>. A follow-up commit cleared the line-length wrap in golden-path.spec.ts.

2. Web E2E was racing the React Query cache. The original golden-path spec navigated to /playground via the SPA nav link. The WorkspaceDetailPage had already populated useKnowledgeBases(workspaceUid) with an empty list (the KB was created via the request fixture, invisible to the page's QueryClient). The cached empty list rode into the playground; the KB select stayed disabled. Fixed by hard-loading /playground via page.goto, which remounts the React app and clears the cache — comment in the spec explains the rationale.

(Java Runtime is a pre-existing failure on main — Spring Boot 4.0.6 missing spring-boot-test-autoconfigure — and is out of scope for this PR.)

Decisions baked in (resolved in the issue thread)

1. KB row carries vector_collection (auto-provisioned) plus lexical_* fields. Lexical isn't a callable service.
2. Embedding service id immutable post-create — vectors on disk are bound to it.
3. Agent's reranking_service_id overrides KB's at query time.
4. Conversations partition-clustered created_at DESC so list returns newest-first.
5. saved_queries does not survive into the new schema.
6. wb_api_key_* unchanged; orthogonal to the data model.

Test plan

• Runtime: `npm run typecheck` clean
• Runtime: `npm run test` — 466/466 passing
• Web UI: `npx tsc -p tsconfig.app.json --noEmit` clean
• Web UI: `npm test` — 76/76 passing
• Conformance: 5 workspace-only scenarios pass; KB scenarios deferred to follow-up
• E2E (`golden-path.spec.ts`) rewritten + green for the KB flow — service creation via API, then KB + playground via UI
• Biome: no errors (8 pre-existing warnings)
• Real-Astra smoke test
• KB conformance scenarios (cross-runtime)

🤖 Generated with Claude Code