From 3b4d0020a7a9e955a3693d2a22649504f6c2f1d2 Mon Sep 17 00:00:00 2001 From: Azeez Ishaqui Date: Sat, 4 Jul 2026 16:52:39 -0400 Subject: [PATCH 1/3] tests: coverage-enforcing serve-matrix release gate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit run_all_models.py boots each model×quant and writes per-model JSON but never exits non-zero on failure — so "verified iff the matrix passes" was a human reading a table, and a model that crashed at boot (no JSON written) was invisible. A glob-only scorer would report the survivors green: a false ship. - run_all_models.py now writes all_models_results/_manifest.json — the roster it planned to cover, derived from planned_specs() (SSOT for the roster). - gate_results.py reads that manifest and fails any planned model with no passing result. Missing manifest is a hard FAIL unless --allow-missing-manifest is passed (explicit, coverage-unenforced). Exit 0 ships, 1 blocks. - tests/test_gate_results.py (stdlib unittest, no GPU) locks in the bars and the planned-but-absent = FAIL property. --- tests/gate_results.py | 225 +++++++++++++++++++++++++++++++++++++ tests/run_all_models.py | 58 ++++++++++ tests/test_gate_results.py | 137 ++++++++++++++++++++++ 3 files changed, 420 insertions(+) create mode 100644 tests/gate_results.py create mode 100644 tests/test_gate_results.py diff --git a/tests/gate_results.py b/tests/gate_results.py new file mode 100644 index 000000000..1e687e7af --- /dev/null +++ b/tests/gate_results.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Serve-matrix release gate — turn single_gpu_suite.py outputs into a PASS/FAIL. + +`tests/run_all_models.py` boots each model×quant and writes one +`tests/all_models_results/