test: set up Cypress visual diff smoke tests#2707
Conversation
|
|
||
| xhr.open('HEAD', url) | ||
| xhr.send() | ||
| }) |
There was a problem hiding this comment.
This helper function wasn't working as expected with the Cypress global afterEach hook so I replaced it with Cypress' native request method.
fac4cbf to
bafb782
Compare
billhimmelsbach
left a comment
There was a problem hiding this comment.
Looking good! 🎉 One comment about how the screenshots are taken, but otherwise super excited to use this in some bug squashing!
56d0a38 to
3eee6c0
Compare
|
@billhimmelsbach I updated the |
There was a problem hiding this comment.
Thanks for making those changes we chatted about this morning! 🎉 Erroring when a screenshot can't be taken definitely helps. I did find two tests that don't have a visual component, so they always fail when taking the baseline.
Rate Spread API
- Generates rates from file
Check Digit API
- Generates Check Digit file
They both just check for 200s, so they both fail. If you want to skip them with a tag, account for them in the snapshot command, or just make a ticket to maybe fix them later if we want to make a CI test, whatever works for ya.
Very excited for this one!🚀
EDIT: I also took a look at running the documentation e2e tests too, and they work well except maybe two things for a future ticket.
- There's the sticky nav that appears bunch on some of the screenshots that we could probably remove?
- Sometimes cypress runs out of memory when comparing the images from tests that include infinite scrolls through the fig content (45mb images).
Something for a future ticket, but this was super exciting to use!
EDIT 2: Something for another issue as we get more things working against staging, but when running against staging, tests that are skipped on staging still have screenshot attempts that fail.

EDIT 3: Looks like maybe the maps are tricky since they can be in various states of loading. I guess for a future ticket/pr we could add more wait or do something different for pages that may take a while to load.
Updates a few resolutions and merges some dependabot PRs. ## Changes - jspdf from 4.2.0 to 4.2.1 - rollup from 4.48.1 to 4.59.0 - flatted from 3.3.3 to 3.4.2 - immutable from 4.3.7 to 4.3.8 - tar from 7.5.9 to 7.5.11 - serialize-javascript from 5.0.1 to 7.0.3 - minimatch from 10.2.1 to 10.2.3 ## Testing 1. Does it look good on staging? Yes! Currently up as `5495-security-updates` 2. Do the PR tests still pass? Yes! 3. Do the e2e tests pass against staging? Mostly yes! Testing against staging is a little tricky, but tests are generally passing except ones that don't work on staging (some filing tests that aren't enabled on the test account, graph tests, and the older mlar tests. We'll work on them soon on [this ticket](https://github.cfpb.gov/HMDA-Operations/hmda-devops/issues/5507)). I also ran the visual regression tests which all mostly passed except a few I noted over here [in this comment](#2707 (review)), which was super exciting 🚀 Closes GHE #5495
3eee6c0 to
7eedc21
Compare
|
Thanks @billhimmelsbach! I added 7eedc21 which skips tests that are tagged |

Adds a global
afterEachhook that runs after every test and takes a screenshot of the app's current state. Baseline screenshots are generated withyarn test-visual:baselineand comparisons against the baseline are made withyarn test-visual.Changes
cypress-visual-regressionlibraryafterEachhookTesting
yarn test-visual:baselineto generate baseline screenshots.yarn test-visualto take new screenshots and compare against the baselines. It should fail if you changed something and pass if you didn't.Notes
actualsubdirectory and check thebasesubdirectory into git. Because this is a new thing and all our environments are a little different, I decided to ignore everything for now.