Skip to content

feat: hash-guarded dataset versioning + release scaffolding #minor#68

Open
BradenBug wants to merge 4 commits into
mainfrom
bw/dataset-versioning
Open

feat: hash-guarded dataset versioning + release scaffolding #minor#68
BradenBug wants to merge 4 commits into
mainfrom
bw/dataset-versioning

Conversation

@BradenBug

@BradenBug BradenBug commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Implements the framework half of the benchmark versioning

What

  • Dataset versioning module (dataset_versioning.py): a dataset_versions.yaml maps each dataset to a human-assigned semver + a sha256 of its content file. load_verified_dataset_versions recomputes and raises DatasetVersionError on any mismatch; a check/update CLI lets maintainers verify and re-stamp checksums.
  • Startup guard (base.py): an opt-in dataset_versions_file ClassVar; when set, create() verifies content before load_datasets() so a service refuses to start on tampered data, and get_dataset_version() serves the declared version.
  • /version field: VersionResponse gains dataset_version, populated from get_dataset_version(dataset) via an optional ?dataset= query param. (Kept on the public /version route deliberately — the version is low-sensitivity metadata already published in manifests.)
  • Scaffolder: create-benchmark-service now emits check-pr-title.yaml + auto-tag-release.yaml and switches scaffolded pyproject.toml to hatch-vcs, so new services get release machinery for free.

@assert-app

assert-app Bot commented Jun 16, 2026

Copy link
Copy Markdown

Review on Assert →

@BradenBug BradenBug marked this pull request as ready for review June 19, 2026 00:13

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

View 2 additional findings in Devin Review.

Open in Devin Review

Comment thread src/benchmark_service/dataset_versioning.py Outdated
Comment thread cli/generator.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant