Skip to content

Use Blacksmith as macOS CI default #5019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lawrencecchen wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Use Blacksmith as macOS CI default #5019

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
63e8f17
Use Blacksmith as macOS CI default
lawrencecchen May 30, 2026
493d3cb
Merge remote-tracking branch 'origin/main' into feat-blacksmith-ci
lawrencecchen May 30, 2026
56156bd
Document macOS runner capacity override
lawrencecchen May 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions .github/actionlint.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,5 @@ self-hosted-runner:
- blacksmith-6vcpu-macos-15
- blacksmith-6vcpu-macos-26
- blacksmith-6vcpu-macos-latest
- warp-macos-15-arm64-6x
- warp-macos-26-arm64-6x
- depot-macos-latest
- depot-macos-14
2 changes: 1 addition & 1 deletion .github/workflows/build-ghosttykit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ concurrency:

jobs:
build-ghosttykit:
runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
timeout-minutes: 20
env:
GHOSTTYKIT_CRASH_REPORT_SUBDIR: cmux/crash
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci-macos-compat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ jobs:
fail-fast: false
matrix:
include:
- os: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
- os: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
timeout: 30
startup_smoke: true
virtual_display: true
skip_zig: false
- os: ${{ vars.MACOS_RUNNER_26 || 'warp-macos-26-arm64-6x' }}
- os: ${{ vars.MACOS_RUNNER_26 || 'blacksmith-6vcpu-macos-26' }}
timeout: 30
startup_smoke: true
virtual_display: false
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ jobs:
bun test tests/vm-db-read-model.test.ts

tests:
runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
timeout-minutes: 75
env:
CMUX_SKIP_ZIG_BUILD: "1"
Expand Down Expand Up @@ -395,7 +395,7 @@ jobs:
# Keep lag validation separate from UI regressions so functional UI failures
# and performance regressions stay isolated. Broader interactive UI suites
# still run via test-e2e.yml on GitHub-hosted runners.
runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
# A cold DerivedData cache (any project.pbxproj or Package.resolved change
# mints a new cache key with no restore-keys fallback) forces a full
# cmux build whose Swift codegen alone runs ~18-20 min, so 20 was right at
Expand Down Expand Up @@ -599,7 +599,7 @@ jobs:
# Keep this on macOS 15 until Zig can link the real universal Ghostty
# CLI helper on macOS 26 runners. The macOS 26 compatibility job still
# covers the app build path with CMUX_SKIP_ZIG_BUILD=1.
runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
timeout-minutes: 20
steps:
- name: Checkout
Expand Down Expand Up @@ -703,7 +703,7 @@ jobs:
lipo "$HELPER_BINARY" -verify_arch arm64 x86_64

ui-regressions:
runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
# Cold builds after project/package changes can spend more than 25 minutes
# in build-for-testing before the UI regression script starts.
timeout-minutes: 45
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ jobs:
build-sign-notarize-nightly:
needs: decide
if: needs.decide.outputs.should_build == 'true'
runs-on: ${{ vars.MACOS_RUNNER_26 || 'warp-macos-26-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_26 || 'blacksmith-6vcpu-macos-26' }}
timeout-minutes: 20
steps:
- name: Checkout build ref
Expand Down
10 changes: 4 additions & 6 deletions .github/workflows/perf-activation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ on:
required: false
default: ""
runner:
description: macOS runner (auto follows the MACOS_RUNNER_15 repo variable, then warp)
description: macOS runner (auto follows the MACOS_RUNNER_15 repo variable, then Blacksmith)
required: false
default: auto
type: choice
Expand All @@ -18,8 +18,6 @@ on:
- blacksmith-6vcpu-macos-15
- blacksmith-6vcpu-macos-26
- blacksmith-6vcpu-macos-latest
- warp-macos-15-arm64-6x
- warp-macos-26-arm64-6x
workspace_count:
description: Fixture workspace count
required: false
Expand All @@ -39,7 +37,7 @@ concurrency:

jobs:
activation-session:
runs-on: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}
runs-on: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}
timeout-minutes: 45
env:
PERF_TAG: perfci
Expand Down Expand Up @@ -94,8 +92,8 @@ jobs:
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: .ci-source-packages
key: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}-${{ hashFiles('cmux.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved') }}
restore-keys: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}-
key: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}-${{ hashFiles('cmux.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved') }}
restore-keys: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}-

- name: Resolve Swift packages
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ env:

jobs:
build-sign-notarize:
runs-on: ${{ vars.MACOS_RUNNER_26 || 'warp-macos-26-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_26 || 'blacksmith-6vcpu-macos-26' }}
timeout-minutes: 20
steps:
- name: Checkout
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test-depot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ on:

jobs:
tests:
runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
timeout-minutes: 20
steps:
- name: Checkout
Expand Down
18 changes: 8 additions & 10 deletions .github/workflows/test-e2e.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: E2E test with video recording
run-name: ${{ inputs.test_filter }} on ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }} @ ${{ inputs.ref || github.ref_name }}
run-name: ${{ inputs.test_filter }} on ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }} @ ${{ inputs.ref || github.ref_name }}

on:
workflow_dispatch:
Expand All @@ -21,7 +21,7 @@ on:
default: true
type: boolean
runner:
description: "Runner OS (auto follows the MACOS_RUNNER_15 repo variable, then warp; pick depot-macos-* for GUI activation)"
description: "Runner OS (auto follows the MACOS_RUNNER_15 repo variable, then Blacksmith; pick depot-macos-* for GUI activation)"
required: false
default: "auto"
type: choice
Expand All @@ -30,26 +30,24 @@ on:
- blacksmith-6vcpu-macos-15
- blacksmith-6vcpu-macos-26
- blacksmith-6vcpu-macos-latest
- warp-macos-15-arm64-6x
- warp-macos-26-arm64-6x
- depot-macos-latest
- depot-macos-14

concurrency:
group: e2e-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}-${{ inputs.ref || github.ref_name }}-${{ inputs.test_filter }}
group: e2e-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}-${{ inputs.ref || github.ref_name }}-${{ inputs.test_filter }}
cancel-in-progress: true

jobs:
e2e:
runs-on: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}
runs-on: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}
timeout-minutes: 20
env:
TEST_REF: ${{ inputs.ref || github.ref }}
steps:
- name: Validate Depot runner identity
if: ${{ startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner, 'depot-macos-') }}
if: ${{ startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner, 'depot-macos-') }}
env:
REQUESTED_RUNNER: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}
REQUESTED_RUNNER: ${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}
RUNNER_CONTEXT_NAME: ${{ runner.name }}
run: |
set -euo pipefail
Expand Down Expand Up @@ -232,8 +230,8 @@ jobs:
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: .ci-source-packages
key: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}-${{ hashFiles('cmux.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved') }}
restore-keys: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner }}-
key: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}-${{ hashFiles('cmux.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved') }}
restore-keys: spm-${{ (!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner }}-

- name: Resolve Swift packages
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/tmux-corpus.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:

terminal-nightly:
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}
runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}
timeout-minutes: 30
steps:
- name: Checkout
Expand Down
20 changes: 11 additions & 9 deletions docs/macos-ci-runners.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,13 @@
All paid macOS CI/CD jobs pick their runner from two repository variables instead of a hardcoded label:

- `MACOS_RUNNER_15` for macOS 15 jobs (most jobs, plus the e2e/perf defaults)
- `MACOS_RUNNER_26` for macOS 26 jobs (release, nightly, the `release-build` job, compat)
- `MACOS_RUNNER_26` for macOS 26 jobs (release, nightly, compat)

Workflows reference them as `runs-on: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }}`. If a variable is unset, the job falls back to WarpBuild, so CI is never broken by a missing variable.
Workflows reference them as `runs-on: ${{ vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15' }}`. If a variable is unset, the job still uses Blacksmith.

## Switch Blacksmith <-> WarpBuild
## Blacksmith defaults

The switch is a repo-variable change. It takes effect on the next workflow run, with no PR or commit.
The defaults are checked into the workflows. The repo variables are an override layer and take effect on the next workflow run, with no PR or commit.

Use Blacksmith (default):

Expand All @@ -18,25 +18,27 @@ gh variable set MACOS_RUNNER_15 --repo manaflow-ai/cmux -b blacksmith-6vcpu-maco
gh variable set MACOS_RUNNER_26 --repo manaflow-ai/cmux -b blacksmith-6vcpu-macos-26
```

Fall back to WarpBuild (e.g. Blacksmith macOS capacity is queuing, as happened in https://github.com/manaflow-ai/cmux/pull/4926):
Remove the override and use the checked-in Blacksmith defaults:

```bash
gh variable delete MACOS_RUNNER_15 --repo manaflow-ai/cmux
gh variable delete MACOS_RUNNER_26 --repo manaflow-ai/cmux
```
Comment on lines +21 to 26
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps Bot May 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 No documented escape hatch for Blacksmith capacity outages

The PR description explicitly cites a past event where Blacksmith queues hit 55–126 minutes and caused a full revert (PR #4926). The old docs told operators to gh variable set … warp-macos-15-arm64-6x; that guidance is now gone, and Warp is removed from all workflow dropdowns and the actionlint allowlist. If Blacksmith queues surge again, an operator in a time-sensitive release has no documented runner to fall back to. The only option—setting MACOS_RUNNER_15/MACOS_RUNNER_26 to a different provider—is correct in principle, but the docs no longer say what labels are valid alternatives or where to find them.

Consider adding a brief "Emergency override" section that names at least one alternative runner label (e.g., a Warp or GitHub-hosted fallback) and notes that the label must also be added to .github/actionlint.yaml before use.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!


Deleting the variables reverts to the WarpBuild fallback baked into the workflows. You can also set them explicitly to `warp-macos-15-arm64-6x` / `warp-macos-26-arm64-6x`.

Check current values:

```bash
gh variable list --repo manaflow-ai/cmux
```

## Capacity events

If Blacksmith queues spike, temporarily point `MACOS_RUNNER_15` and `MACOS_RUNNER_26` at another paid macOS runner label with `gh variable set`, then restore the Blacksmith values after the incident. Do not change checked-in workflow defaults for a temporary capacity event.

## Manual runs

`perf-activation.yml` and `test-e2e.yml` keep a `runner` choice input that defaults to `auto`. `auto` (and the empty `pull_request` case for perf) follows `MACOS_RUNNER_15` then the Warp fallback, so flipping the repo variable also redirects these workflows. An explicit choice wins over the variable; both dropdowns expose `warp-macos-15-arm64-6x` / `warp-macos-26-arm64-6x` so an operator can pick Warp directly during a Blacksmith outage. `test-e2e.yml` also keeps `depot-macos-*` choices and a Depot identity guard for GUI-activation runs.
`perf-activation.yml` and `test-e2e.yml` keep a `runner` choice input that defaults to `auto`. `auto` (and the empty `pull_request` case for perf) follows `MACOS_RUNNER_15` then the checked-in Blacksmith default, so flipping the repo variable also redirects these workflows. An explicit choice wins over the variable. `test-e2e.yml` also keeps `depot-macos-*` choices and a Depot identity guard for GUI-activation runs.

## Guard

`tests/test_ci_self_hosted_guard.sh` (run by the `workflow-guard-tests` job) asserts every paid macOS job references `vars.MACOS_RUNNER_*` or a Blacksmith/Warp label, so a job can never silently fall back to a free GitHub-hosted runner. Keep new labels in `.github/actionlint.yaml`.
`tests/test_ci_self_hosted_guard.sh` (run by the `workflow-guard-tests` job) asserts every paid macOS job references `vars.MACOS_RUNNER_*` or a Blacksmith label, so a job can never silently fall back to a free GitHub-hosted runner. Keep new labels in `.github/actionlint.yaml`.
12 changes: 6 additions & 6 deletions tests/test_ci_self_hosted_guard.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
# Regression test for https://github.com/manaflow-ai/cmux/issues/385.
# Ensures paid CI jobs use a paid macOS runner (Blacksmith or WarpBuild, routed
# Ensures paid CI jobs use a paid macOS runner (Blacksmith, routed
# through the MACOS_RUNNER_15 / MACOS_RUNNER_26 repo variables), never a free
# GitHub-hosted runner. Flip Blacksmith<->Warp by editing those repo variables;
# GitHub-hosted runner. Override Blacksmith by editing those repo variables;
# see docs/macos-ci-runners.md.
# Fork PRs are gated by GitHub's built-in "Require approval for outside
# collaborators" setting, so workflow-level fork guards are not needed.
Expand All @@ -19,11 +19,11 @@ check_macos_runner() {
if ! awk -v job="$job" '
$0 ~ "^ "job":" { in_job=1; next }
in_job && /^ [^[:space:]#][^:]*:[[:space:]]*(#.*)?$/ { in_job=0 }
in_job && /runs-on:.*(vars\.MACOS_RUNNER|blacksmith-[0-9]+vcpu-macos-|warp-macos-[0-9]+-arm64)/ { saw=1 }
in_job && /os:.*(vars\.MACOS_RUNNER|blacksmith-[0-9]+vcpu-macos-|warp-macos-[0-9]+-arm64)/ { saw=1 }
in_job && /runs-on:.*(vars\.MACOS_RUNNER|blacksmith-[0-9]+vcpu-macos-)/ { saw=1 }
in_job && /os:.*(vars\.MACOS_RUNNER|blacksmith-[0-9]+vcpu-macos-)/ { saw=1 }
END { exit !(saw) }
' "$file"; then
echo "FAIL: $job in $(basename "$file") must run on a paid macOS runner (vars.MACOS_RUNNER_* or a Blacksmith/Warp label), not a GitHub-hosted runner"
echo "FAIL: $job in $(basename "$file") must run on a paid macOS runner (vars.MACOS_RUNNER_* or a Blacksmith label), not a GitHub-hosted runner"
exit 1
fi
echo "PASS: $job in $(basename "$file") uses a paid macOS runner"
Expand Down Expand Up @@ -61,7 +61,7 @@ check_e2e_runner_fallbacks() {
exit 1
fi

if ! grep -Fq "startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x') || inputs.runner, 'depot-macos-')" "$E2E_FILE"; then
if ! grep -Fq "startsWith((!inputs.runner || inputs.runner == 'auto') && (vars.MACOS_RUNNER_15 || 'blacksmith-6vcpu-macos-15') || inputs.runner, 'depot-macos-')" "$E2E_FILE"; then
echo "FAIL: test-e2e.yml must validate all Depot macOS runner choices"
exit 1
fi
Expand Down
Loading