Skip to content

ci(windows): bust vcpkg cache on overlay/triplet/requirements changes#5934

Open
Fedr wants to merge 1 commit intomasterfrom
ci/windows-vcpkg-cache-key
Open

ci(windows): bust vcpkg cache on overlay/triplet/requirements changes#5934
Fedr wants to merge 1 commit intomasterfrom
ci/windows-vcpkg-cache-key

Conversation

@Fedr
Copy link
Copy Markdown
Contributor

@Fedr Fedr commented Apr 19, 2026

Problem

The Windows vcpkg cache in .github/workflows/build-test-windows.yml is keyed only by matrix.vcpkg-version:

key: vcpkg-cache-${{ matrix.vcpkg-version }}

Changes to any of these don't invalidate the cache:

  • thirdparty/vcpkg/ports/** — MeshLib's overlay ports (e.g. custom opencascade-minimal, laz-perf, openctm, clip, and as of #5932 a zlib → zlib-ng-compat override)
  • thirdparty/vcpkg/triplets/** — custom triplets passed via --overlay-triplets
  • requirements/windows.txt — the package list install.bat feeds to vcpkg
  • thirdparty/install.bat — changes to how vcpkg is invoked

A pre-built C:\vcpkg\installed\ tree gets restored verbatim; install.bat's vcpkg install ... step sees every transitively-required port as already installed and completes in a few milliseconds without consulting overlay ports.

Evidence from PR #5932's Windows run

From windows-build-test (msvc-2022, Release, CMake, …) on run 24629386911:

12:46:32  Cache hit for: vcpkg-cache-2026.03.18
12:48:11  Cache restored from key: vcpkg-cache-2026.03.18
          ...
          The following packages are already installed:
              boost-*, freetype[core,zlib,...], libzip[core,...], tiff[zip,...], ...
          Total install time: 3.57 ms

zlib doesn't even appear in the already-installed list (it's a transitive dep). vcpkg saw every port satisfied by the cached tree — which had been populated on master before PR #5932's overlay existed — and declared the install plan done. The overlay was detected (Using overlay: D:\a\MeshLib\MeshLib\thirdparty\vcpkg\ports appears in the log) but never used because nothing needed installing.

Compression benchmark numbers, for reference (from the MRMesh.CompressSphereToZip test in PR #5933):

master (gzip) PR #5932 branch (zlib-ng) Δ
linux-vcpkg Release x64 Clang 2441 ms 1336 ms −45.3%
linux-vcpkg Release arm64 Clang 1467 ms 860 ms −41.4%
windows-build-test Release CMake msvc-2022 2182 ms 2115 ms −3.1% (noise)
windows-build-test Release CMake msvc-2019 2163 ms 2210 ms +2.2% (noise)

Linux-vcpkg rebuilds vcpkg from scratch inside the prepare-image / linux-vcpkg-build-upload Docker image each time, so it doesn't hit this cache-key problem. Windows did.

Fix

Expand the cache key to hash the relevant paths:

key: vcpkg-cache-${{ matrix.vcpkg-version }}-${{ hashFiles('thirdparty/vcpkg/ports/**', 'thirdparty/vcpkg/triplets/**', 'requirements/windows.txt', 'thirdparty/install.bat') }}

Any change to overlay ports, triplets, the package list, or the install script busts the cache and forces a fresh install. Steady-state unchanged: same branch twice in a row still hits the cache.

Cost

First run after this lands does the full Windows vcpkg install (~20 min, one-time). Subsequent same-branch runs hit the cache again as before. Every future overlay-affecting PR pays a one-time rebuild, which is the correct behavior.

Not in this PR

Rollback

Single-file revert of .github/workflows/build-test-windows.yml restores the old cache key. No lingering state — the cache entries under the new key just become orphans that expire naturally.

The Windows vcpkg cache key was only keyed by matrix.vcpkg-version,
so changes to:
  - thirdparty/vcpkg/ports/**   (MeshLib's overlay ports)
  - thirdparty/vcpkg/triplets/**
  - requirements/windows.txt
  - thirdparty/install.bat
didn't invalidate the cache. A pre-built C:\vcpkg\installed\ tree
was restored verbatim and install.bat's "vcpkg install ..." saw
every port as "already installed" (total install time: 3.57 ms in
the last PR run), so overlay ports never got consulted.

Concrete symptom observed on PR #5932 (zlib → zlib-ng-compat overlay):
  - Linux-vcpkg (which rebuilds vcpkg inside a Docker image every
    time `prepare-image / linux-vcpkg-build-upload` runs) picked up
    zlib-ng; compression tests showed -0.3% zip size and -41 to -45%
    wall time on Release builds.
  - Windows vcpkg served the cached pre-zlib-ng tree; zip sizes and
    test times were indistinguishable from master.

Fix: include a hashFiles() of the overlay/triplet/requirements/
install-script content in the cache key. First run after merging
this takes the full ~20 min vcpkg install; subsequent runs hit the
cache again as before.
Fedr added a commit that referenced this pull request Apr 20, 2026
Observed in run 24638998286: on Windows, a fresh install of our
zlib overlay (triggered by PR #5934 busting the vcpkg cache key)
failed with:

  error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED
  Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst
  ... 404 from every msys2 mirror

Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on
Windows invokes vcpkg_acquire_msys to download the pinned
mingw-w64-pkgconf package from msys2 mirrors. That particular version
has been pruned from the live mirrors; every one of the six fallback
URLs 404s. Deterministic failure that re-runs won't fix.

The call is not needed by our consumers. Every MeshLib/vcpkg consumer
that depends on zlib finds it via the CMake target `ZLIB::ZLIB`
exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads
lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated
by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat
mode - it just doesn't get post-processed.

Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat
build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup
for ZLIB, copyright install) are unaffected.
Fedr added a commit that referenced this pull request Apr 20, 2026
One-line change to .github/workflows/build-test-windows.yml: append a
unique suffix to the vcpkg cache key so the existing cached
C:\vcpkg\installed\ tree is not restored. vcpkg reinstalls every
port from scratch; upstream blosc's vcpkg_fixup_pkgconfig() tries to
fetch mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst from msys2
mirrors, gets 404 on all six, and fails the Windows build.

This is the same failure first seen on PR #5932 after PR #5934 (which
adds the path-based cache busting in a principled way) exposed it.
Master itself doesn't fail because its Windows vcpkg cache is a hot
hit every run.

Purely a diagnostic PR. Do not merge.
MaxRayskiy pushed a commit that referenced this pull request Apr 21, 2026
Observed in run 24638998286: on Windows, a fresh install of our
zlib overlay (triggered by PR #5934 busting the vcpkg cache key)
failed with:

  error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED
  Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst
  ... 404 from every msys2 mirror

Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on
Windows invokes vcpkg_acquire_msys to download the pinned
mingw-w64-pkgconf package from msys2 mirrors. That particular version
has been pruned from the live mirrors; every one of the six fallback
URLs 404s. Deterministic failure that re-runs won't fix.

The call is not needed by our consumers. Every MeshLib/vcpkg consumer
that depends on zlib finds it via the CMake target `ZLIB::ZLIB`
exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads
lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated
by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat
mode - it just doesn't get post-processed.

Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat
build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup
for ZLIB, copyright install) are unaffected.
MaxRayskiy pushed a commit that referenced this pull request Apr 21, 2026
Observed in run 24638998286: on Windows, a fresh install of our
zlib overlay (triggered by PR #5934 busting the vcpkg cache key)
failed with:

  error: building zlib:x64-windows-meshlib failed with: BUILD_FAILED
  Missing msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst
  ... 404 from every msys2 mirror

Root cause: our portfile called vcpkg_fixup_pkgconfig(), which on
Windows invokes vcpkg_acquire_msys to download the pinned
mingw-w64-pkgconf package from msys2 mirrors. That particular version
has been pruned from the live mirrors; every one of the six fallback
URLs 404s. Deterministic failure that re-runs won't fix.

The call is not needed by our consumers. Every MeshLib/vcpkg consumer
that depends on zlib finds it via the CMake target `ZLIB::ZLIB`
exported from lib/cmake/ZLIB/ZLIBConfig.cmake; nobody reads
lib/pkgconfig/zlib.pc via pkg-config. The .pc file is still generated
by zlib-ng's own CMake, with a correct `Libs: -lz` entry in compat
mode - it just doesn't get post-processed.

Removing the call sidesteps the msys2 dependency entirely. zlib-ng-compat
build steps (cmake configure, cmake install, copy_pdbs, cmake_config_fixup
for ZLIB, copyright install) are unaffected.
Fedr added a commit that referenced this pull request Apr 23, 2026
Resolves conflicts from master's recent work (MRZlib move MRIOExtras→
MRMesh, #5948 adds MRZlibTests.cpp, #5933+#5960 overhaul CompressZip
tests, #5957 CUDA cache-key fix, #5722 triplet-keyed vcpkg cache).

Conflict resolutions:

  - .github/workflows/build-test-windows.yml — combined both sides'
    improvements in the cache key: master-side per-triplet partition
    (for the x64-windows-meshlib-iterator-debug triplet from #5722)
    AND branch-side hashFiles(overlay ports / triplets / requirements
    / install.bat) (from #5934, needed so overlay port changes on this
    PR bust the cache). Key is now
    vcpkg-cache-<VERSION>-<TRIPLET>-<HASH>.

  - source/MRMesh/MRZlib.h — accept master's version (master re-homed
    the file in #5944; branch had deleted it when it lived briefly in
    MRIOExtras).

  - source/MRIOExtras/MRZlib.{h,cpp} — follow master's layout: keep
    MRIOExtras/MRZlib.h as a forwarding header (#include
    <MRMesh/MRZlib.h>) and delete MRIOExtras/MRZlib.cpp (the old
    implementation body, now in MRMesh).

  - source/MRTest/MRTest.vcxproj — rename ClCompile entry from
    MRZlib.cpp to MRZlibTests.cpp to match master's #5948 test-file
    rename. Also drops the duplicate MRZlib.cpp entry.

  - source/MRTest/MRZlib.cpp — deleted, master renamed to
    MRZlibTests.cpp in #5948.

  - source/MRTest/MRZipCompressTests.cpp — accept master's version,
    which includes the level-0..9 loop (#5960) and the round-trip
    decompress check (#5958) on top of the branch's earlier simpler
    version. Branch-side novelty was just the sphere test itself,
    which is already on master.

  - source/MRTest/MRZlibTests.cpp — accept master's version,
    engine-agnostic assertions (tolerance-based compressedSize checks
    that I relaxed in the zlib-ng PR work).

Branch's unique contributions — the actual point of this PR — are
preserved:

  - thirdparty/vcpkg/ports/zlib/{portfile.cmake,vcpkg.json}  -- vcpkg
    overlay port that builds zlib-ng in ZLIB_COMPAT=ON as a drop-in
    zlib replacement for every port that depends on zlib.
  - thirdparty/install.bat  -- --overlay-ports flag added alongside
    the existing --overlay-triplets.
  - thirdparty/vcpkg/downloads/msys2-*.pkg.tar.zst  -- pre-staged
    msys2 packages so vcpkg's pinned 2024.10.21 index doesn't hit
    stale msys2 mirrors during vcpkg_fixup_pkgconfig().
Fedr added a commit that referenced this pull request Apr 23, 2026
The master-merge earlier on this branch brought in PR #5722's
cache-key format (vcpkg-cache-<VERSION>-<TRIPLET>) but dropped the
hashFiles() appendix from #5934. On this PR that matters: the last
run (24830925083) had a cache hit on the older key and reused a pre-
overlay vcpkg installed tree, so "vcpkg install" saw every port
already-installed and built zero of this PR's overlay zlib port.
Effect: MRMesh.dll and libzip ended up linked against stock zlib,
and the CompressSphereToZip numbers on Windows matched master's
stock-zlib baseline (1172-1355 ms Release) instead of the zlib-ng
speedup we see on Linux-vcpkg legs in the same run (~450 ms Release).

Re-add the hashFiles() over thirdparty/vcpkg/{ports,triplets}/,
requirements/windows.txt, and thirdparty/install.bat so a change to
any of them busts the cache and forces a fresh install. On this PR
specifically, that ensures the new thirdparty/vcpkg/ports/zlib
overlay actually gets built and replaces stock zlib with zlib-ng-
compat in the cached tree.

The fix for the unit-test failure (MRMesh.ZlibCompressStats pinned
stats.compressedSize against stock-zlib reference byte counts) came
in automatically with the master merge -- master's version of
source/MRTest/MRZlibTests.cpp uses engine-agnostic assertions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant