Skip to content

Normalize snippet quality scoring: pass_rate + stable-only gating + artifact upload#863

Draft
Josverl with Copilot wants to merge 8 commits into
mainfrom
copilot/update-workflow-success-calculation
Draft

Normalize snippet quality scoring: pass_rate + stable-only gating + artifact upload#863
Josverl with Copilot wants to merge 8 commits into
mainfrom
copilot/update-workflow-success-calculation

Conversation

Copilot AI commented Feb 24, 2026

Copy link
Copy Markdown
Contributor

The snippet quality workflow was gating PRs on an absolute passed - failed score, making results highly sensitive to skip volatility, version churn, and stale baselines. A 5% real quality drop could be masked or a mass-skip could look fine.

Changes

conftest.py — normalized score metrics

  • Adds executed = passed + failed and pass_rate = passed / executed to snippet_score.json
  • Writes the file whenever executed > 0 (was: only when snippet_score > 0)
  • Keeps snippet_score for backward compatibility
  • Adds --stable-only pytest CLI flag to filter preview versions from test parameterization

test_snippets.py — stable-only version filtering

  • Renames VERSIONSALL_VERSIONS
  • In pytest_generate_tests, respects --stable-only by excluding versions ending with -preview

compare_score.py — tolerant, normalized comparison

PASS_RATE_TOLERANCE = 0.05   # allow up to 5% drop before failing
MIN_EXECUTED_FRACTION = 0.50  # require ≥50% of baseline executed count
  • Checks minimum executed count first (catches mass-skip / environment failure)
  • Compares pass_rate within tolerance instead of raw score delta
  • Falls back to legacy snippet_score comparison when baseline predates this change

test_stub_quality.yml — split gating and preview jobs

  • test_snippets (gating): adds --stable-only; uploads results as snippet-results-stable-{run_id}
  • test_snippets_preview (non-blocking): new job, continue-on-error: true, runs all versions; uploads as snippet-results-preview-{run_id}
  • Both jobs set permissions: contents: read

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI changed the title [WIP] Update workflow to clarify success criteria based on derived scores Normalize snippet quality scoring: pass_rate + stable-only gating + artifact upload Feb 24, 2026
Copilot AI and others added 2 commits February 24, 2026 22:00
…able-only gating

Co-authored-by: Josverl <981654+Josverl@users.noreply.github.com>
@Josverl Josverl force-pushed the copilot/update-workflow-success-calculation branch from 2139f7b to 98c9e4f Compare February 24, 2026 21:00
Josverl and others added 6 commits February 24, 2026 22:16
…ased on --stable-only flag

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
…ison in CI workflow

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
…enhance multi-version support

Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Signed-off-by: Jos Verlinde <Jos_Verlinde@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants