Skip to content

test: set up Cypress visual diff smoke tests#2707

Open
contolini wants to merge 5 commits intomasterfrom
5269-visual-smoke
Open

test: set up Cypress visual diff smoke tests#2707
contolini wants to merge 5 commits intomasterfrom
5269-visual-smoke

Conversation

@contolini
Copy link
Copy Markdown
Member

Adds a global afterEach hook that runs after every test and takes a screenshot of the app's current state. Baseline screenshots are generated with yarn test-visual:baseline and comparisons against the baseline are made with yarn test-visual.

Changes

Testing

  1. Run yarn test-visual:baseline to generate baseline screenshots.
  2. Change some aspect of the app.
  3. Run yarn test-visual to take new screenshots and compare against the baselines. It should fail if you changed something and pass if you didn't.

Notes

  • I gitignored the entire screenshots directory. Normally you'd only ignore the actual subdirectory and check the base subdirectory into git. Because this is a new thing and all our environments are a little different, I decided to ignore everything for now.


xhr.open('HEAD', url)
xhr.send()
})
Copy link
Copy Markdown
Member Author

@contolini contolini Mar 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This helper function wasn't working as expected with the Cypress global afterEach hook so I replaced it with Cypress' native request method.

Copy link
Copy Markdown
Contributor

@billhimmelsbach billhimmelsbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! 🎉 One comment about how the screenshots are taken, but otherwise super excited to use this in some bug squashing!

Comment thread cypress/support/e2e.js Outdated
@contolini contolini force-pushed the 5269-visual-smoke branch 3 times, most recently from 56d0a38 to 3eee6c0 Compare March 31, 2026 19:05
@contolini
Copy link
Copy Markdown
Member Author

@billhimmelsbach I updated the afterEach hook to fail the assertion if the screenshot element can't be found on the page, see 3eee6c0. You can test it locally by changing the selectors to gibberish and yarn test-visual:baseline --spec=cypress/e2e/common-features/navigation.spec.js.

@contolini contolini enabled auto-merge March 31, 2026 19:08
Copy link
Copy Markdown
Contributor

@billhimmelsbach billhimmelsbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making those changes we chatted about this morning! 🎉 Erroring when a screenshot can't be taken definitely helps. I did find two tests that don't have a visual component, so they always fail when taking the baseline.

Rate Spread API
  - Generates rates from file

Check Digit API
  - Generates Check Digit file

They both just check for 200s, so they both fail. If you want to skip them with a tag, account for them in the snapshot command, or just make a ticket to maybe fix them later if we want to make a CI test, whatever works for ya.

Very excited for this one!🚀

EDIT: I also took a look at running the documentation e2e tests too, and they work well except maybe two things for a future ticket.

  1. There's the sticky nav that appears bunch on some of the screenshots that we could probably remove?
  2. Sometimes cypress runs out of memory when comparing the images from tests that include infinite scrolls through the fig content (45mb images).

Something for a future ticket, but this was super exciting to use!

EDIT 2: Something for another issue as we get more things working against staging, but when running against staging, tests that are skipped on staging still have screenshot attempts that fail.
Screenshot 2026-04-02 at 2 44 44 PM

EDIT 3: Looks like maybe the maps are tricky since they can be in various states of loading. I guess for a future ticket/pr we could add more wait or do something different for pages that may take a while to load.

From a screenshot as an example:
Screenshot 2026-04-02 at 3 09 40 PM

billhimmelsbach added a commit that referenced this pull request Apr 6, 2026
Updates a few resolutions and merges some dependabot PRs.

## Changes

- jspdf from 4.2.0 to 4.2.1
- rollup from 4.48.1 to 4.59.0 
- flatted from 3.3.3 to 3.4.2 
- immutable from 4.3.7 to 4.3.8 
- tar from 7.5.9 to 7.5.11 
- serialize-javascript from 5.0.1 to 7.0.3 
- minimatch from 10.2.1 to 10.2.3

## Testing

1. Does it look good on staging? Yes! Currently up as `5495-security-updates`
2. Do the PR tests still pass? Yes!
3. Do the e2e tests pass against staging? Mostly yes!
Testing against staging is a little tricky, but tests are generally passing except ones that don't work on staging (some filing tests that aren't enabled on the test account, graph tests, and the older mlar tests. We'll work on them soon on [this ticket](https://github.cfpb.gov/HMDA-Operations/hmda-devops/issues/5507)). I also ran the visual regression tests which all mostly passed except a few I noted over here [in this comment](#2707 (review)), which was super exciting 🚀

Closes GHE #5495
@contolini
Copy link
Copy Markdown
Member Author

Thanks @billhimmelsbach! I added 7eedc21 which skips tests that are tagged @no-visual and I created ticket #5537 to track the above changes that need to be made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants