From 20ab4887fc8ac6cab4829b2f0783d23e69d9bf74 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 28 Jun 2026 20:09:01 -0400 Subject: [PATCH 1/6] New (draft) post: Opus 4.8 on GHA-bench Draft post (published: false) summarizing the opus-4.8 GHA-bench results: quality leader on tests + code (panel agrees, Spearman +0.83/+0.90) at a speed/cost premium (steepest at medium ~+65% vs 4.7, compressing to ~+15% at high; D/D- at xhigh/ultra), the new multi-agent "ultra" effort, and the trap caveat (~2x traps but ~99% benign per the 201-occurrence forensic). Notes the agy judge-harness and CC-version caveats. Links the updated sorting widget. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Qx8jvUUZeg6DY1nxBT4pKf --- _posts/2026-06-28-opus-4-8-on-gha-bench.md | 45 ++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 _posts/2026-06-28-opus-4-8-on-gha-bench.md diff --git a/_posts/2026-06-28-opus-4-8-on-gha-bench.md b/_posts/2026-06-28-opus-4-8-on-gha-bench.md new file mode 100644 index 000000000..7e0a1f3e2 --- /dev/null +++ b/_posts/2026-06-28-opus-4-8-on-gha-bench.md @@ -0,0 +1,45 @@ +--- +title: "Opus 4.8 on GHA-bench: the new quality leader, at a price" +slug: opus-4-8-on-gha-bench +date: 2026-06-28 12:00:00 -0400 +excerpt: Anthropic's Opus 4.8 is now the strongest model on GHA-bench at writing + and testing GitHub Actions — and also the slowest and most expensive. Here's the + shape of the tradeoff. +featured_image: /assets/images/uploads/img_9581.png +published: false +--- +[GHA-bench](https://github.com/Adam-S-Daniel/GHA-bench) — my benchmark for how well coding agents author and test GitHub Actions — now includes Anthropic's **Opus 4.8**. The short version: it's the best model I've measured at this task, and also the slowest and priciest. Here's the shape of the tradeoff. + +The run is a full sweep: 7 tasks × 5 scripting languages × 4 effort levels = 140 agent runs, each graded by a panel of judges (Google Gemini and Claude Haiku) on test comprehensiveness and code quality. + +## Opus 4.8 tops the quality charts + +Across the board, Opus 4.8 produces the strongest **tests** and the strongest **deliverable code** of any model in the benchmark. At high, xhigh, and the new "ultra" effort it earns A‑/A grades for test quality in nearly every language — for example A (4.6) on the default language at ultra effort, and A (4.5) for PowerShell at high. The two judges, despite coming from different labs, agree on the ranking (Spearman +0.83 on test quality, +0.90 on code quality), so this isn't one judge's quirk. + +## …but you pay for it + +The flip side is time and money. Relative to Opus 4.7, the premium is steepest at **medium** effort — roughly **+65% on both wall-clock time and cost** — and, interestingly, *compresses to about +15% at high effort* (4.7's "high" is comparatively expensive, so 4.8 closes the gap). At the top two efforts (xhigh and "ultra"), individual runs routinely take **15–25 minutes and cost $4.50–$7.35**, landing in the D / D‑ bands on the speed and cost curves. + +If you want most of Opus 4.8's quality without the worst of the bill, **medium effort is the value sweet spot**: B+/B‑ on speed, C+/C on cost, and still A‑/B+ on test quality in several languages. + +## The new "ultra" effort + +This run introduces a fourth effort level — **"ultra"** — which layers multi-agent orchestration on top of the highest reasoning setting. It tops the test-quality charts (it's the single best column for tests) but is the most expensive option on the board, and it's Opus‑4.8‑only, so there's no older-model baseline to compare it against yet. Treat it as "spend more for the most thorough tests," not as a free win. + +## It iterates a lot — but it isn't getting stuck + +Opus 4.8 writes **more and denser tests** than its predecessor, and it shows: it also trips GHA-bench's "trap" detectors (heuristics that flag things like re-running the same test command many times) about **twice as often** as Opus 4.7. That sounds alarming, so I hand-reviewed **all 201** of those occurrences. The result: + +- **99% show no looping at all.** +- **86%** are legitimate engineering — red-green TDD cycles, designing fixtures up front, fixing a real type error — that merely tripped a count-based heuristic. +- **~1%** looked like genuine distress. + +So read "~2× the traps" as **"iterates ~2× more granularly," not "fails ~2× as often."** A good chunk of the gap is also a measurement artifact (the way 4.8 prefixes its shell commands defeats the detector's de-duplication) plus a Claude Code version difference between the two runs, not the model spinning its wheels. The [full investigation](https://github.com/Adam-S-Daniel/GHA-bench/blob/main/results/analysis/opus48-trap-investigation_2026-06-28.md) has the details. + +## Try it yourself + +The interactive sorting widget in [the original GHA-bench post](/introducing-gha-bench) now includes Opus 4.8 (and the new "ultra" effort). Drag the sliders to weight speed, cost, test quality, and code quality for *your* situation and see which model / effort / language combination comes out on top. The complete data lives in the [cross-run report](https://github.com/Adam-S-Daniel/GHA-bench/tree/main/results) on GitHub. + +*\* The Gemini judge now runs via Google's Antigravity (`agy`) CLI, which replaced the retired Gemini CLI in June 2026. It grades about 0.3 points stricter on a 1–5 scale than the prior harness (overall correlation r ≈ 0.90), so Opus 4.8's quality grades are, if anything, very slightly conservative relative to the older models'.* + +*\*\* The two runs being compared used different Claude Code versions (2.1.131/132 for Opus 4.7, 2.1.195 for Opus 4.8). A clean, model-only comparison would re-run both on the same version; that's on the to-do list.* From ec0120b40783a93a028c3aac9267bfa9c0c97685 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 28 Jun 2026 20:14:47 -0400 Subject: [PATCH 2/6] Embed the sorting widget (with presets) directly in the Opus 4.8 post Replace the link-only 'Try it yourself' section with the full interactive best-weighted-sorting table (Opus 4.8 + comparison models, one pooled calibration) plus the five weighting presets. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01Qx8jvUUZeg6DY1nxBT4pKf --- _posts/2026-06-28-opus-4-8-on-gha-bench.md | 314 ++++++++++++++++++++- 1 file changed, 312 insertions(+), 2 deletions(-) diff --git a/_posts/2026-06-28-opus-4-8-on-gha-bench.md b/_posts/2026-06-28-opus-4-8-on-gha-bench.md index 7e0a1f3e2..ba309fcf0 100644 --- a/_posts/2026-06-28-opus-4-8-on-gha-bench.md +++ b/_posts/2026-06-28-opus-4-8-on-gha-bench.md @@ -36,9 +36,319 @@ Opus 4.8 writes **more and denser tests** than its predecessor, and it shows: it So read "~2× the traps" as **"iterates ~2× more granularly," not "fails ~2× as often."** A good chunk of the gap is also a measurement artifact (the way 4.8 prefixes its shell commands defeats the detector's de-duplication) plus a Claude Code version difference between the two runs, not the model spinning its wheels. The [full investigation](https://github.com/Adam-S-Daniel/GHA-bench/blob/main/results/analysis/opus48-trap-investigation_2026-06-28.md) has the details. -## Try it yourself +## Which should you use? Try it yourself -The interactive sorting widget in [the original GHA-bench post](/introducing-gha-bench) now includes Opus 4.8 (and the new "ultra" effort). Drag the sliders to weight speed, cost, test quality, and code quality for *your* situation and see which model / effort / language combination comes out on top. The complete data lives in the [cross-run report](https://github.com/Adam-S-Daniel/GHA-bench/tree/main/results) on GitHub. +Pick a preset, or drag the sliders to weight speed, cost, test quality, and code quality for *your* situation — the table re-ranks every model / effort / language combination live. (Same widget as in [the original GHA-bench post](/introducing-gha-bench), now with Opus 4.8 and the new "ultra" effort.) + + +
+
+
+ Presets: + + + + + +
+
+
+ + + 17.5% +
+
+ + + 17.5% +
+
+ + + 40.0% +
+
+ + + 25.0% +
+
+ + + + + + + + + + + + +
ModelLanguageDurationCostTestsCode
+
+ + + + +
+ + +The complete data lives in the [cross-run report](https://github.com/Adam-S-Daniel/GHA-bench/tree/main/results) on GitHub. *\* The Gemini judge now runs via Google's Antigravity (`agy`) CLI, which replaced the retired Gemini CLI in June 2026. It grades about 0.3 points stricter on a 1–5 scale than the prior harness (overall correlation r ≈ 0.90), so Opus 4.8's quality grades are, if anything, very slightly conservative relative to the older models'.* From 0005dfc2530d14083daaf67dd33052e5cc257798 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 6 Jul 2026 08:59:47 -0400 Subject: [PATCH 3/6] Refresh Opus 4.8 draft for the geomean report overhaul (PR-38 numbers) Widget regenerated from the current 6-run combined report: 76 rows, geometric-mean duration/cost, powershell-tool pooled into pwsh, and sonnet 5 + fable 5 + split opus-4.6/sonnet-4.6 effort rows added. Duration cell now shows a slowest-run tooltip with timeout censoring (dagger marker + footnote). Prose: cost premiums re-derived (+54%/+67% at medium, ~+30% at high, ~+75-80% at xhigh), Spearman correlation range updated, and a new "how to read the numbers" methodology paragraph added. --- _posts/2026-06-28-opus-4-8-on-gha-bench.md | 159 ++++++++++++--------- 1 file changed, 92 insertions(+), 67 deletions(-) diff --git a/_posts/2026-06-28-opus-4-8-on-gha-bench.md b/_posts/2026-06-28-opus-4-8-on-gha-bench.md index ba309fcf0..ffdf00fca 100644 --- a/_posts/2026-06-28-opus-4-8-on-gha-bench.md +++ b/_posts/2026-06-28-opus-4-8-on-gha-bench.md @@ -12,15 +12,17 @@ published: false The run is a full sweep: 7 tasks × 5 scripting languages × 4 effort levels = 140 agent runs, each graded by a panel of judges (Google Gemini and Claude Haiku) on test comprehensiveness and code quality. +**How to read the numbers:** duration and cost figures here are *geometric* means — outlier-damped, so one unusually slow run can't dominate a cell — and runs that hit the 30-minute timeout count against the duration statistics at their recorded wall clock (a ≥ marks a measurement the timeout capped). The tables also pool the benchmark's two PowerShell variants (script-file vs. inline-tool) into a single `pwsh` column; in this harness they turned out to be replicates of each other. + ## Opus 4.8 tops the quality charts -Across the board, Opus 4.8 produces the strongest **tests** and the strongest **deliverable code** of any model in the benchmark. At high, xhigh, and the new "ultra" effort it earns A‑/A grades for test quality in nearly every language — for example A (4.6) on the default language at ultra effort, and A (4.5) for PowerShell at high. The two judges, despite coming from different labs, agree on the ranking (Spearman +0.83 on test quality, +0.90 on code quality), so this isn't one judge's quirk. +Across the board, Opus 4.8 produces the strongest **tests** and the strongest **deliverable code** of any model in the benchmark — including the newer Sonnet 5 and Fable 5 runs that have joined the dataset since this comparison was first drawn up. At high, xhigh, and the new "ultra" effort it earns A‑/A grades for test quality in nearly every language — for example A (4.6) on the default language at ultra effort, and A (4.5) for PowerShell at high. The two judges, despite coming from different labs, agree on the ranking (Spearman rank correlations of +0.66 to +0.85 across the model and language-by-model rankings), so this isn't one judge's quirk. ## …but you pay for it -The flip side is time and money. Relative to Opus 4.7, the premium is steepest at **medium** effort — roughly **+65% on both wall-clock time and cost** — and, interestingly, *compresses to about +15% at high effort* (4.7's "high" is comparatively expensive, so 4.8 closes the gap). At the top two efforts (xhigh and "ultra"), individual runs routinely take **15–25 minutes and cost $4.50–$7.35**, landing in the D / D‑ bands on the speed and cost curves. +The flip side is time and money. Relative to Opus 4.7, the premium is steepest at **medium** effort — roughly **+54% on wall-clock time and +67% on cost** — and, interestingly, *compresses to about +30% at high effort* (4.7's "high" is comparatively expensive, so 4.8 closes part of the gap) before re-widening to about +75–80% at xhigh. At the top two efforts (xhigh and "ultra"), typical runs take **15–25 minutes and cost $4.30–$6.95** — with the slowest hitting the 30-minute timeout cap — landing in the D / D‑ bands on the speed and cost curves. -If you want most of Opus 4.8's quality without the worst of the bill, **medium effort is the value sweet spot**: B+/B‑ on speed, C+/C on cost, and still A‑/B+ on test quality in several languages. +If you want most of Opus 4.8's quality without the worst of the bill, **medium effort is the value sweet spot**: B+ on speed in three of the four languages, C+/C on cost, and still A‑ test quality everywhere except bash. ## The new "ultra" effort @@ -38,7 +40,7 @@ So read "~2× the traps" as **"iterates ~2× more granularly," not "fails ~2× a ## Which should you use? Try it yourself -Pick a preset, or drag the sliders to weight speed, cost, test quality, and code quality for *your* situation — the table re-ranks every model / effort / language combination live. (Same widget as in [the original GHA-bench post](/introducing-gha-bench), now with Opus 4.8 and the new "ultra" effort.) +Pick a preset, or drag the sliders to weight speed, cost, test quality, and code quality for *your* situation — the table re-ranks every model / effort / language combination live. (Same widget as in [the original GHA-bench post](/introducing-gha-bench), now with Opus 4.8, the new "ultra" effort, and the Sonnet 5 and Fable 5 runs that have landed since. Hover or long-press a Duration cell to see that combination's slowest run; † marks combos where a run hit the 30-minute timeout.)
@@ -86,6 +88,9 @@ Pick a preset, or drag the sliders to weight speed, cost, test quality, and code +

† at least one run in this combination hit the 30-minute + timeout cap; its slowest-run figure (hover/long-press the Duration cell) is a lower + bound, shown with ≥.