diff --git a/_posts/2026-05-12-introducing-gha-bench.md b/_posts/2026-05-12-introducing-gha-bench.md index 090fdcd2b..8c27347ff 100644 --- a/_posts/2026-05-12-introducing-gha-bench.md +++ b/_posts/2026-05-12-introducing-gha-bench.md @@ -13,15 +13,25 @@ published: true Agents (currently a variety of Anthropic models set to various effort levels, driven by Claude Code) are given [set of tasks](https://github.com/Adam-S-Daniel/GHA-bench/blob/main/benchmark-instructions-v4.md#tasks) they must automate using GitHub Actions, either using a particular scripting language or whichever they want.\* They must use Test-Driven Development (TDD)-- basically "write tests first, and don't come back until they all pass".\** -A panel of judges (Google Gemini and Claude Haiku) then [evaluates](https://github.com/Adam-S-Daniel/GHA-bench/blob/main/AGENTS.md#:~:text=Evaluate%20test%20%2B%20deliverable%20quality) the comprehensiveness of the tests and the quality of the code. +A panel of judges (Google Gemini and Claude Haiku) then [evaluates](https://github.com/Adam-S-Daniel/GHA-bench/blob/main/AGENTS.md#:~:text=Evaluate%20test%20%2B%20deliverable%20quality) the comprehensiveness of the tests and the quality of the code.\*** ## Which model, effort level and scripting language should you use? -Adjust the sliders according to your priorities. +The table below now includes **Opus 4.8** — at medium, high, and xhigh effort, plus a new "ultra" effort that layers in multi-agent orchestration — alongside Opus 4.7, Sonnet 4.6, Opus 4.6, Sonnet 5, Fable 5, and Haiku 4.5, with the two PowerShell variants pooled into a single `pwsh` column. Every row is graded on a single shared curve pooled across all runs, so the letter grades are comparable across models. + +Pick a preset, or adjust the sliders yourself.