Skip to content

Weekly-pipeline cleanup: remove deprecated workflow and follow-ups - #272

Open
BenjSz wants to merge 5 commits into
mainfrom
benjams/remove_deprecated_action
Open

Weekly-pipeline cleanup: remove deprecated workflow and follow-ups#272
BenjSz wants to merge 5 commits into
mainfrom
benjams/remove_deprecated_action

Conversation

@BenjSz

@BenjSz BenjSz commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Remove deprecated workflow. Delete `.github/workflows/test_weekly_DEPRECATED.yaml` — a Saturday-scheduled job that just printed the top-10 embedding-model report to logs. Its purpose has been fully absorbed by the ClickHouse-backed pipeline in `push-to-clickhouse.yaml` + `tests/spyre/weekly_generation/`.
  • Rename `generate-matrix` → `generate-shards` in `push-to-clickhouse.yaml`. That job's real output is a set of shard files uploaded as an artifact; the `matrix_x{1,2,4}` JSON it emits is just an index into that artifact. The old name confusingly overlapped with the actual matrix-generator jobs of the same name in `test_daily.yaml` / `_test_matrix.yaml`.
  • Clean up docstrings in `push-to-clickhouse.yaml`: rewrite the top-of-file overview to describe what the fetch/shard/fan-out flow actually does, and drop the stale reference to the deleted `test_weekly_DEPRECATED.yaml`.
  • Accept `DD/MM/YYYY` dates in `tests/spyre/weekly_generation/clickhouse_db.py`. `import_csv` used to reject rows whose `snapshot_date`/`added_date` came from a European-formatted CSV export. A new shared `_parse_date` helper accepts ISO first, then a few day-first fallbacks — ambiguous US month-first strings are intentionally not accepted.
  • Drop dead plot code in `tests/spyre/weekly_generation/snapshot_model_counts.py`: the two commented-out `_plot_1`/`_plot_2` blocks and the disabled funnel-bar section are gone. `_plot_3` is now the only plot function, renamed to `_plot_models`, and its "Three-panel" docstring corrected to "Two-panel".

Test plan

  • `gh workflow list` no longer shows the deprecated `test_weekly` workflow.
  • Confirm no other workflow references the deleted file (verified locally: `git grep test_weekly_DEPRECATED` returns nothing).
  • Manually dispatch `push-to-clickhouse.yaml` with a small `top_k` to confirm the `generate-shards` rename didn't break the `needs:` wiring for the three `weekly-model-scan-x{1,2,4}` jobs.
  • Run `snapshot_model_counts.py` against an enriched CSV to confirm the plot still renders after the `_plot_3 → _plot_models` rename.
  • Run `clickhouse_db.py --add_csv` against a CSV containing a `DD/MM/YYYY` `snapshot_date` and confirm the row imports without hitting the "invalid snapshot_date" branch.

🤖 Generated with Claude Code

Signed-off-by: Benjamin Sznajder <benjams@il.ibm.com>
@BenjSz
BenjSz requested a review from kmehant as a code owner August 3, 2026 10:58
@BenjSz
BenjSz enabled auto-merge August 3, 2026 11:20
@BenjSz
BenjSz requested a review from assaftibm August 3, 2026 11:20
BenjSz and others added 3 commits August 3, 2026 14:40
- push-to-clickhouse.yaml: rewrite the top-of-file docstring to describe
  what generate-matrix actually does (sharding job that emits an index
  into an uploaded artifact), and drop the reference to the now-removed
  test_weekly_DEPRECATED.yaml.
- clickhouse_db.py: import_csv previously rejected rows whose
  snapshot_date/added_date came from a CSV using European DD/MM/YYYY
  formatting. Add a shared _parse_date helper that accepts ISO plus a few
  day-first variants, so those CSVs round-trip without silent data loss.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Benjamin Sznajder <benjams@il.ibm.com>
The job's real output is a set of shard files uploaded as an artifact;
the matrix_x{1,2,4} JSON it emits is just an index into that artifact,
not a matrix generator like the same-named jobs in test_daily.yaml /
_test_matrix.yaml. Rename the job (and all needs: references in this
file) to match what it actually does, and simplify the top-of-file
docstring now that the "misnamed" caveat is gone.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Benjamin Sznajder <benjams@il.ibm.com>
Remove the two commented-out _plot_1/_plot_2 blocks and the disabled
funnel-bar section, and rename _plot_3 to _plot_models now that it's
the only plot function. Also correct the "Three-panel" docstring to
"Two-panel" — the third panel was already gone.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Benjamin Sznajder <benjams@il.ibm.com>
@BenjSz
BenjSz requested review from arielge and kiszk as code owners August 3, 2026 13:07
@BenjSz BenjSz changed the title Remove deprecated test_weekly workflow Weekly-pipeline cleanup: remove deprecated workflow and follow-ups Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant