fix(e2e): exclude test fixtures from @parity content crawls + robust ephemeral loop timing (#1771 follow-up) - #1809
Merged
Conversation
…ephemeral loop timing (#1771 follow-up) The #1771 step-4 ephemeral prod loops stayed red in real prod due to a vicious cycle + a #1723 Cat-2 regression. Root cause (proven locally): the ephemeral prod-loop posts are created through the genuinely-UI-driven Decap "+ New Post" leg. Decap writes only the posts-collection fields, which declare NO sitemap/robots and whose test_fixture is a hidden default:false widget the editor can't toggle. So the post that lands on main carries published:true + a future date + test_fixture:false and NO sitemap/robots keys (verified against the real Create Post commits) — it looks exactly like real public content. The @parity content-crawl specs (console-clean in required e2e-admin AND parity; image-alt-text; sitemap) enumerated it, and a transient orphan a crashed run left on main (deleted image now 404s) red-failed those required checks on EVERY cms PR — including the loop's own create PR, so the loop could never go green. The design's intended test_fixture:true / sitemap:false markers (only the composePost afterAll fallback sets them) were never actually on the UI-created posts, so a test_fixture-only exclusion would not have worked. Fix A (primary — breaks the cycle): a single shared, well-named enumeration predicate e2e/public-content.js#isTestFixturePost, used by console-clean, image-alt-text, and sitemap. It excludes a post on ANY of: test_fixture:true OR sitemap:false (the _e2e/unpublish canaries) OR the structural e2e- slug signature (/^\d{4}-\d{2}-\d{2}-e2e-/ filename, ^e2e- URL slug — the codebase-wide fixture detector). The slug signature is the load-bearing one: it catches the UI-created posts that carry neither flag. Locked by e2e/public-content.test.js (incl. the UI-shaped no-flag post + a console-clean-style end-to-end enumeration). Local negative-control proof: with the exclusion disabled both console-clean and image-alt-text FAIL on a UI-shaped canary's 404'd resource / no-alt img; with it on, they never enumerate it. Fix B (robust timing): reopenForPublishedDelete 6 -> 13 min (it timed out waiting for Decap to re-sync to published after the create PR's squash auto-merge + branch removal); create-PR waitForMerge 5 -> 10 min (API lag absorption). TEST_TIMEOUT_MS raised to fit the job timeouts (prod 40->46 min within the 50-min job; media 55->80 min within the 95-min job). Fix C (orphan sweeping): sweep-stale-cms-prs.yml gains a dedicated ephemeral_threshold_hours (default 3h, vs 6h human-hostable tiers; 3h is deliberately above the longest run's 95-min job timeout so the sweep cannot reap a post still under active test) and a 3-hourly cron, so a killed run's orphan is gone within ~3-6h. Docs: reconciled the inaccurate "born with test_fixture:true" claims in AGENTS.md, prod-mutate-fixture.js, and cms-posts-list-enhance.spec.js to reflect what the UI actually writes, and documented the Cat-2 regression + public-content crawl scope. select-specs.js wires public-content.js as a dependency of the three crawl specs + the new unit test. All constraint lints green (silent-catch, select-specs, cms-recursion-churn, cms-editor-ui, fixture-baseline, canary-content, workflow-prod-loop-serialized); npm run lint + format:check clean. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
🗑️ Preview environment cleaned up. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why the loops stayed red (the vicious cycle + #1723 Cat-2 regression)
The #1771 step-4 ephemeral prod loops (
cms-publish-loop-prod-mutate,cms-media-roundtrip) were still failing in real prod after #1797/#1801/#1804. The proven root cause, reproduced locally:The ephemeral posts are created through the genuinely-UI-driven Decap "+ New Post" leg. Decap writes only the fields the
postscollection declares (admin/config*.yml), which has nositemap/robotsfield and whosetest_fixtureis awidget: hidden, default: falsethe editor can't toggle. So the post that actually lands onmaincarriespublished: true+ a future date +test_fixture: falseand nositemap/robotskeys — verified against the realCreate Postcommit:It therefore looks exactly like real public content. The
@paritycontent-crawl specs enumerate every published/blog/post:console-clean.spec.jsruns in BOTH the requirede2e-admin(@admin-read) andparitychecks;image-alt-text.spec.jscrawls every sitemap URL (the post lands in the sitemap — nositemap: false).When a crashed run leaves an orphan on
main(a post whose uploaded image was already deleted),/blog/<slug>/404s a resource → console-clean/image-alt-text red-fail → every cms PR is BLOCKED, including the loop's OWN create PR (verified: create PR #1808 BLOCKED one2e-admin/parity/finalizefailing on/blog/e2e-media-roundtrip-…/). The create PR never merges → the post never serves → the loop's serve leg times out → the run fails → leaves another orphan → repeat. This is the exact #1723 Category-2 class (transientmainstate poisoning a shared required check), reintroduced by the ephemeral redesign.Crucially, the design's intended
test_fixture: true/sitemap: falsemarkers exist only in thecomposePostafterAll fallback — they were never on the UI-created posts — so atest_fixture-only exclusion would not have worked.Fix A (primary — breaks the cycle): public-content crawls exclude test fixtures
A single shared, well-named enumeration predicate,
e2e/public-content.js#isTestFixturePost, is now the source of truth used byconsole-clean,image-alt-text, andsitemap. It excludes a post on ANY of:test_fixture: true(the documented marker —_e2e/unpublish canaries, composePost fallback),sitemap: false(fixtures opt out of the sitemap),e2e-slug signature (/^\d{4}-\d{2}-\d{2}-e2e-/filename,^e2e-URL slug — already the codebase-wide fixture detector inadmin/posts-list-enhance.js).Signal 3 is load-bearing: it catches the UI-created posts that carry neither flag, because the
slug:/date:the spec types are reliably present. Real public posts (even one titled "E2E Testing Guide" with a non-e2e-slug) are not excluded.Locked by
e2e/public-content.test.js, including the exact UI-shaped no-flag post and a console-clean-style end-to-end enumeration.Local proof (negative control): with the exclusion temporarily disabled, both
console-clean(a 5th test for the canary URL →console.error: Failed to load resource … 404) andimage-alt-text(<img>with no alt on the canary) FAIL on a UI-shaped canary that is in the built sitemap and renders at/blog/…/. With the exclusion on, the canary URL is never enumerated and both stay green.Fix B: robust ephemeral create→serve→delete→404 timing
reopenForPublishedDelete6 → 13 min (it timed out waiting for Decap to re-sync to the published file after the create PR's squash auto-merge + branch removal; under contention 6 min is too tight).waitForMerge5 → 10 min (absorbs GitHub API lag between deploy completing and the PR flippingmerged:true; the merge has normally already landed by the URL-serve gate).TEST_TIMEOUT_MSraised to fit the worst-case leg sum within the job timeouts: prod 40 → 46 min (50-min job), media 55 → 80 min (95-min job).The loop still genuinely exercises the real Decap UI create→publish→serve→UI-delete→404 chain; the afterAll existence-only API safety-net is unchanged.
Fix C: harden orphan sweeping
sweep-stale-cms-prs.ymlgains a dedicatedephemeral_threshold_hours(default 3h) for the ephemeral prod-loop orphan tier (vs 6h for human-hostable tiers) plus a 3-hourly cron, so a killed run's orphan is gone within ~3-6h. 3h is deliberately above the longest run's 95-min job timeout so the sweep can never reap a post still under active test. With Fix A this is defence-in-depth (orphans no longer poison checks) + tidiness.Docs / lint plumbing
test_fixture: true" claims inAGENTS.md,prod-mutate-fixture.js, andcms-posts-list-enhance.spec.jsto reflect what the UI actually writes; documented the Cat-2 regression + the public-content crawl scope.select-specs.jswirese2e/public-content.jsas a dependency of the three crawl specs (and the parity-preview selector) + maps the new unit test.Validation
npm run lint(0 errors) +npm run format:checkclean.--project chromium-desktop-3k --list.TARGET=localruns of console-clean / image-alt-text / sitemap / draft-isolation pass with AND without a UI-shaped canary present.🤖 Generated with Claude Code