Skip to content

ci(deploy): gate API deploy on affected paths - #2711

Open
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/fd758nco3oiw
Open

ci(deploy): gate API deploy on affected paths#2711
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/fd758nco3oiw

Conversation

@polylane

@polylane polylane Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Fixes: Content-only commit redeploys API + all Vercel apps; web build flaky

Every push to main currently redeploys the production API, even a commit that only edits marketing copy or the README. That means a content typo can churn the API fleet and its healthchecks for no reason, and every merge multiplies deploy risk across four production surfaces. This change adds a paths filter to the Fly Deploy workflow so the API deploys only when apps/server or one of the packages it depends on changes, mirroring the gate the workflows app already uses.

What caused this

Affected: int_01b1b9dbe001vd8itqqr5fpc · severity medium

What changed

  • .github/workflows/deploy.yml: added an on.push.paths filter so the API deploy runs only when apps/server/**, a dependency package under packages/** (analytics, assertions, db, emails, error, importers, locales, notifications, proto, regions, services, subscriptions, theme-store, tinybird, tracker, tsconfig, upstash, utils), the workflow file itself, or the pnpm manifests change.

Why it's safe

  • Missed-deploy risk: the filter is checked against the API's actual build surface (apps/server/Dockerfile bind-mount list and package.json dependencies); every mounted/imported package is covered, plus pnpm-lock.yaml/pnpm-workspace.yaml and the workflow file itself.
  • Behavior parity: the gate mirrors the exact structure the sibling deploy-workflows.yml has run in production since before this incident; no other workflow semantics are touched.
  • Undershoot vs overshoot: package dirs the API does not consume (ai, api, header-analysis, icons, react, status-fetcher, test-utils, ui) are deliberately absent, so the filter does not reintroduce the noise it removes.
  • Content commits are the only skipped class: anything that changes server code, a dependency package, or the lockfile still deploys exactly as before.

Validation

  • YAML: parsed with ruby's YAML loader (YAML.load_file succeeds; the only quirk is YAML 1.1 treating on as boolean, which GitHub's Actions parser handles and every workflow in this repo shares).
  • Structure: verified against the live sibling .github/workflows/deploy-workflows.yml — same quoting, same branches/paths layout, same indentation.
  • Filter coverage: cross-checked against apps/server/Dockerfile bind-mount list and apps/server/package.json dependencies — every package the API image mounts or imports is in the filter.
  • not run: pnpm verify (repo lint/typecheck): the change is a workflow YAML only; no TS/JS or package code changed, so oxfmt/oxlint/deno check have nothing to evaluate in this diff.
Root cause and scoping notes

Root cause

  • Symptom: 2026-09-13 19:30Z commit a6887e1 (PR chore: improve agent-first marketing visibility #2710, 10 files, all README + apps/web content/metadata) triggered an openstatus-api redeploy (release 2210, complete 19:30:42Z) and production builds in all three Vercel projects.
  • Mechanism: .github/workflows/deploy.yml fires on push: branches: [main] with no paths filter, so every main push runs flyctl deploy for the API. The sibling deploy-workflows.yml already gates its app on paths; the API workflow never got the same gate.
  • Vercel half (not fixable by PR): only openstatus-web has commandForIgnoringBuildStep: "npx turbo-ignore" in the Vercel console; dashboard and status-page have none, and no repo file declares these settings. Recorded as a follow-up on the issue timeline.
  • Web build flake: the Turbopack postcss crash is a single occurrence in 7 days; the manual redeploy of the same commit succeeded (dpl_GZAdYq4Qf2Q4rpcYGGTq4cmmz7JY, READY/PROMOTED 20:43Z), so it is not reproduced and no code change is made against it.
  • This change removes the repo-owned producer: after merge, a content-only commit matches no paths and the workflow is skipped.
Causal chain
  • Signal (alert): Content-only commit redeploys API + all Vercel apps; web build flaky
  • Surfacing site: OpenStatus deploy pipeline (GitHub Actions + Vercel + Fly.io) at openstatusHQ/openstatus:.github/workflows/deploy.yml#on.push
  • Mechanism: deploy.yml triggers on push to main with no paths filter, so every main push runs flyctl deploy for openstatus-api regardless of file changes; Vercel projects openstatus-dashboard and openstatus-status-page have no Ignored Build Step configured (only openstatus-web has npx turbo-ignore), so they rebuild on every commit
  • Producer: .github/workflows/deploy.yml (Fly Deploy workflow), instance main-branch push trigger, at openstatusHQ/openstatus:.github/workflows/deploy.yml#on.push
  • Trigger: Push of commit a6887e1 (PR chore: improve agent-first marketing visibility #2710, marketing content only) to main on 2026-09-13 19:30Z
  • What happens to the failed unit today: A failed Vercel build leaves production serving the prior READY build until a human redeploys (thibaultleouay's manual redeploy of the same commit succeeded 20:43Z, ~70 min gap); a failed Fly deploy fails the workflow with no auto-rollback
  • Cadence check: The no-paths-filter trigger deploys the API on every main push (~6-10 commits/day baseline), so release 2210 firing on a content-only commit is exactly what the config predicts; the web build ERROR is 1 occurrence in 7 days and did not recur on the same commit's retry, so it is a flake, not a reproducible defect
  • Blast radius: 3 other resource(s), 0 other tenant(s); data at risk: No data at risk; the exposure is deploy churn and build flakiness across the API plus the three Vercel apps (status-page serves ~255 tenant domains), where a failed build leaves tenants on a stale build
  • Producer evidence:
    • flyListReleases(openstatus-api): release 2210 complete 2026-09-13T19:30:42Z, matching the a6887e1 push
    • githubCommits a6887e1: 10 files, all README.md + apps/web content/metadata, no code
    • Vercel projects API: openstatus-web commandForIgnoringBuildStep='npx turbo-ignore'; openstatus-dashboard and openstatus-status-page both null
    • vercelListDeployments: all three projects show production deploys for a6887e1 at 19:30Z

Detection outcome

The signal "every commit, even a marketing-copy-only change, redeploys the API server" stops firing once this merges: the Fly Deploy workflow is now gated on paths, so a content-only commit produces no API release (the 19:30:42Z release 2210 class of redeploy cannot occur for such a commit). The same commit's rebuilds of openstatus-dashboard and openstatus-status-page are a Vercel-console setting (no ignore-build-step command configured), tracked as a follow-up on the issue timeline rather than fixed here.

Fix chosen

Chosen: cause (removes the mechanism that produces the failure): Add an on.push.paths filter to .github/workflows/deploy.yml so the API deploys only when apps/server, a dependency package, the workflow, or the pnpm manifests change.

Considered and not chosen:

  • cause (removes the mechanism that produces the failure): Split the API deploy into a separate workflow gated on paths, one workflow per app. Not chosen: More machinery than needed; the repo already has the paths-gate convention in deploy-workflows.yml, and applying the same pattern to deploy.yml is the minimal change that removes the producer.
  • cause (removes the mechanism that produces the failure): Add Ignored Build Step npx turbo-ignore to the dashboard and status-page Vercel projects so they skip unaffected commits. Not chosen: The settings are declared only in the Vercel console, which is read-only from this workspace; no repo file owns them. Recorded as a follow-up on the issue timeline for a person with console access.
  • disable (turns a feature, guard, check, test, or telemetry off): Remove the API deploy workflow entirely and deploy manually. Not chosen: Disables a required production deploy path; not a fix, and removes the ability to ship API changes at all.

Outcome after fix

Replaying the 2026-09-13 scenario with the change applied: a push of a6887e1 (README + apps/web/** only) matches none of the paths entries, so GitHub skips Fly Deploy entirely — no openstatus-api release is created, no fleet churn, no healthcheck reset. A push that touches apps/server/**, packages/db/**, or pnpm-lock.yaml matches the filter and the workflow runs byte-for-byte as it does today. The status-page and dashboard production builds on content-only commits will keep firing until the Vercel console setting is applied (recorded on the issue timeline); the API half of the finding is removed by this change.

1 file changed (+23/-0)
  • .github/workflows/deploy.yml: modified, +23/-0

Repository conventions present in the repository: AGENTS.md, CLAUDE.md, apps/checker/AGENTS.md, apps/dashboard/AGENTS.md, apps/server/AGENTS.md, apps/status-page/AGENTS.md, apps/web/AGENTS.md, apps/workflows/AGENTS.md, packages/services/AGENTS.md, packages/ui/AGENTS.md.

Repository lint: pnpm run lint (declared in package.json) could not run in the sandbox because its tool is not installed there; run it before merging.

View autofix View thread


Generated by Polylane.

Review in cubic

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane polylane Bot added the polylane label Sep 14, 2026
@vercel

vercel Bot commented Sep 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
openstatus-dashboard Ready Ready Preview Sep 14, 2026 12:51am UTC
openstatus-status-page Ready Ready Preview Sep 14, 2026 12:51am UTC
openstatus-web Ready Ready Preview Sep 14, 2026 12:51am UTC

Request Review

@polylane

polylane Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

The PR only touches .github/workflows/deploy.yml — a paths filter. The failed checks are Tests, api, and services jobs. Let me look at the CI workflows to understand what those jobs run and where the failures point.

The test workflow runs turbo-filtered tests per package. Let me see the actual test commands and the rest of the workflow, then reproduce the failures locally.

The PR touches only deploy.yml, so the api/services test failures are suspicious — I need the actual CI failure logs rather than guessing from line numbers. Let me fetch the check runs for the commit.

The api leg failed (run 34793719478), and I need the services leg's id and the actual failure text. Let me grep the spill file and fetch annotations plus job logs.

The spill is pretty-printed JSON. Let me parse it with python to get the run ids and conclusions.

The spill file was truncated (65KB shown, 65,841 bytes total, ends mid-JSON). Let me fetch the check runs again with a tight projection directly.

The failing legs are 🧪 api and 🧪 services (plus the aggregate 🧪 Tests). The PR only touches deploy.yml — no code. Let me get the actual failure annotations and job logs.

The annotations confirm @openstatus/api#test and @openstatus/services#test exited 1. Now I need the actual test output. Fetching the job logs.

The job logs are saved to a spill file. Let me search it for the actual failure lines.

The grep matched too much (spill got re-spilled). Let me look at the structure of the file first.

The log is JSON-escaped and the spill was truncated. Let me fetch each log and extract just the failing test section inside the sandbox.

The real failures are in two test files, both unrelated to my deploy.yml change: packages/api/src/router/statusPage.utils.test.ts (setDataByType duration tests, "Expected actual to not be strictly equal to: undefined") and packages/services/src/page-component/__tests__/get-daily-summary.test.ts ("Expected value to have length 2, but it does not: the value has length 1"). Let me reproduce them locally.

The failures are in test files my PR never touched (deploy.yml paths filter). Before touching anything, I need to know whether these tests fail on main too. Let me check recent test.yml runs on main.

The base commit a6887e1 passed Tests on main (run 34777866818 success), and the PR branch is base + one YAML file. Let me read the full failure stacks to confirm the exact assertions before deciding.

The failure picture is now clear from the CI logs I already have: @openstatus/api#test fails in statusPage.utils.test.ts (setDataByType duration tests) and @openstatus/services#test fails at get-daily-summary.test.ts:187 (empty-buckets count). The PR's only change is a YAML paths filter in a deploy workflow, and the identical Tests workflow passed on the base commit a6887e1 on main at 19:30Z. Both failures have a suspicious common factor: CI ran at 00:46Z, right after midnight UTC, and both failing tests build events relative to new Date(). Let me confirm the mechanism in the code.

No deno binary and no services/.env.test in the sandbox, so a local reproduction of the full suites isn't possible here. The failing tests need a DB (services test uses withTestTransaction), and there's no sqld container. But the key question isn't reproduction: it's whether my deploy.yml change could have caused these. It cannot — the diff is a single workflow file. Let me read the two failing tests to nail the mechanism.

The failing assertions are expect(successCard).toBeDefined() and expect(successCard?.value).toBeTruthy() — the success card is missing. Both tests passed on main 5 hours earlier. Let me check the test helpers for a time dependency.

The helpers getAdjustedTotalMinutesInDay and getTotalEventsDurationMs are local to statusPage.utils.ts at lines 143/152. Let me read them and the downtimeIntervals/downtime.ts machinery to nail the exact time-of-day mechanism.

Retained head: the failing CI runs at 00:46Z for api (statusPage.utils.test.ts:530/547, successCard undefined) and services (get-daily-summary.test.ts:187 length 1 vs 2). I read the code path: createStatusData day 0 = "today", events span [00:00, 01:00) UTC, setDataByType duration uses getAdjustedTotalMinutesInDay(date, maintenances) = getTotalMinutesInDay(date) minus maintenance; getTotalMinutesInDay lines 120-126: if date is "today", returns minutesElapsed since midnight (00:00 at 00:46Z), else 24*60. Line 115-119 elided: getTotalMinutesInDay(date) returns minutesElapsed when date.getTime() === startOfDay at midnight? Let me re-read that top part, since the 00:00Z edge is the crux: successMinutes = max(1440 - 60 - 0, 0) = 1380 normally, but at 00:46Z with the minute-elapsed path it could be ~46 - 60 → clamped 0 → null. That's the mechanism, and the same "current day" logic explains services get-daily-summary (2 expected buckets: completed day + in-progress day; at 00:46Z the in-progress day has only elapsed minutes, or the query filters to days with data).

I need: (a) the exact getTotalMinutesInDay head (lines 100-120), (b) the services get-daily-summary day-window logic, to state the mechanism precisely, then decide: fix or no-action. The PR is a YAML-only change; both failures are time-of-day-sensitive tests that passed at 19:30Z on the identical code. My change cannot have caused them. The question is whether a fix belongs in this PR (it does not — the PR is the deploy gate), so I should end with the no-action note unless the failures are flaky in a way that needs a code change unrelated to my diff.

Let me re-read the two code sections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants