Skip to content

RAG: retrieval quality evaluation set #178

Description

@qiuethan

Sub-issue of the RAG retrieval epic (#125).

Scope

  • A small fixed question set with expected source documents, checked into the repo.
  • A runnable harness reporting retrieval hit rate at k, so chunking and model changes can be compared rather than guessed at.
  • Deliberately small and hand-built: enough to catch a regression, not a benchmark suite.

Acceptance

  • The harness runs locally against a seeded corpus and reports a score.
  • Changing chunk size produces a measurably different number.

Why this is in the epic

Wiring the bot (#81) to retrieval before knowing whether retrieval is any good makes bad answers look like a bot problem.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/docs-systemdocumentation-system service & content pipelinearea/toolingDev & docs tooling / automationtype/featureNew capability or enhancement

    Type

    No type

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions