MRZlib: implement zlibCompressStream via libdeflate#5954
Draft
MRZlib: implement zlibCompressStream via libdeflate#5954
Conversation
Only the stats-enabled zlibCompressStream(in, out, const ZlibCompressParams&) overload changes: it now reads the input stream into memory and hands compression + CRC-32 to libdeflate (~20-40% faster than zlib-ng, larger delta in Debug builds since libdeflate's base implementation doesn't depend on runtime SIMD dispatch). The thin int-level overload continues to forward to this one. Decompression (zlibDecompressStream) and the compressZip/libzip pipeline both stay on stock zlib -- this PR is intentionally scoped to the single function the user requested. Tradeoff: libdeflate has no streaming input API, so the compressor materialises the whole input in memory. Memory ceiling = input stream size; callers that compress very large inputs (multi-GB) should chunk above this function. Level mapping: ZlibCompressParams's -1 (zlib default) -> libdeflate 6; ZlibCompressParams's 0 (zlib stored-only) -> libdeflate 1 because libdeflate has no stored-only mode. The existing MRMesh.ZlibCompressStats test allows ±4 bytes around the stock-zlib reference sizes, which is enough to absorb the minor byte-level differences in libdeflate's output. Adds thirdparty/libdeflate submodule pinned at v1.24, built as a shared library (LIBDEFLATE_BUILD_GZIP/STATIC/TESTS off) from thirdparty/ CMakeLists.txt on all platforms. MRMesh finds it via find_package (libdeflate CONFIG) hinted at the thirdparty install prefix.
Windows CI (both MSBuild and CMake legs) doesn't run the thirdparty
CMake build step that builds submodules into the install prefix --
dependencies come from vcpkg via thirdparty/install.bat. So gate the
add_subdirectory(./libdeflate) on NOT WIN32 and add 'libdeflate' to
requirements/windows.txt so vcpkg supplies it there.
All other platforms keep building the submodule-pinned v1.24:
- Ubuntu apt (22.04 ships 1.10, 24.04 ships 1.19 -- both older than
v1.24, so we prefer the pinned submodule build for consistent perf)
- Rocky Linux vcpkg (CMake toolchain, can consume submodule fine)
- macOS Homebrew (brew 1.25 is current but keeping submodule for
version parity across non-Windows legs)
- Emscripten (no package manager)
MRMesh's find_package(libdeflate CONFIG HINTS ...) call is unchanged
and works uniformly: on Windows the vcpkg toolchain's CMAKE_PREFIX_PATH
resolves it; elsewhere the HINTS path at the thirdparty install prefix
picks up our submodule build. Same libdeflate::libdeflate_shared target.
…locatable config)
libdeflate's own libdeflateConfig.cmake is not relocatable: it sets the
exported target's INTERFACE_INCLUDE_DIRECTORIES via $<INSTALL_INTERFACE:
${CMAKE_INSTALL_FULL_INCLUDEDIR}> (absolute path). On MeshLib's Ubuntu
and Emscripten Docker images, thirdparty is built at /home/MeshLib
during the build stage and then COPYed to /usr/local/lib/meshlib-
thirdparty-lib/ in the production stage. The baked-in /home/MeshLib/
include path no longer exists in the production image, so CMake fails
the imported target's path-existence check with:
Imported target "libdeflate::libdeflate_shared" includes non-existent
path "/home/MeshLib/include" in its INTERFACE_INCLUDE_DIRECTORIES
Other thirdparty libs (httplib, nlohmann_json, ...) are fine because
their configs use the relative CMAKE_INSTALL_INCLUDEDIR and stay
relocatable. libdeflate up to at least v1.24 does not.
Switch to plain find_library + find_path. MeshLib's root CMakeLists
already adds ${MESHLIB_THIRDPARTY_ROOT_DIR}/lib and /include to the
link/include search paths, and the vcpkg toolchain on Windows does the
equivalent. Same binary, no config-package indirection, avoids the bug.
MaxRayskiy
force-pushed
the
feat/zlib-compress-stream-libdeflate
branch
from
69fddcb to
594977a
Compare
Fedr
added a commit
that referenced
this pull request
Apr 22, 2026
…locatable config) Same issue and same fix as applied on the sibling PR #5954: libdeflate's libdeflateConfig.cmake is not relocatable -- the exported target's INTERFACE_INCLUDE_DIRECTORIES is set via $<INSTALL_INTERFACE: ${CMAKE_INSTALL_FULL_INCLUDEDIR}> (absolute path). On MeshLib's Ubuntu and Emscripten Docker images, thirdparty is built at /home/MeshLib in the build stage and then COPYed to /usr/local/lib/meshlib-thirdparty- lib/ in the production stage; the baked-in /home/MeshLib/include path no longer exists in the production image, so CMake fails: Imported target "libdeflate::libdeflate_shared" includes non-existent path "/home/MeshLib/include" in its INTERFACE_INCLUDE_DIRECTORIES macOS builds pass because the macOS scripts install thirdparty to the same directory the main build sees; Windows uses vcpkg's own copy so the submodule config is never consulted. Ubuntu (x64+arm64) and Emscripten all trip the path-existence check at CMake configure time. Switch to plain find_library + find_path. The MeshLib root CMakeLists already adds ${MESHLIB_THIRDPARTY_ROOT_DIR}/lib and /include to the link/include search paths, and the vcpkg toolchain on Windows does the equivalent. Same libdeflate.so / libdeflate.h either way.
This was referenced Apr 22, 2026
…lib-compress-stream-libdeflate
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
Exactly one function flips backend:
```cpp
Expected zlibCompressStream( std::istream& in, std::ostream& out, const ZlibCompressParams& params );
```
It now reads the input stream into memory and hands the deflate primitive + CRC-32 to libdeflate. The thin `zlibCompressStream(in, out, int level)` overload forwards to it unchanged. Everything else — `zlibDecompressStream` (both overloads) and the entire `compressZip` / libzip pipeline — stays on stock zlib.
Why
Per public libdeflate benchmarks, its deflate is ~20–40% faster than zlib-ng (which is already ~1.5–2× faster than stock zlib) and achieves a better compression ratio at level 12. Its base implementation doesn't depend on runtime SIMD dispatch, so the speedup is visible in Debug builds too, where zlib-ng's vectorised hot paths often degrade.
Tradeoffs
Build
Adds `thirdparty/libdeflate` submodule pinned at `v1.24`, built as shared-only (`LIBDEFLATE_BUILD_GZIP/STATIC/TESTS` off) unconditionally from `thirdparty/CMakeLists.txt`. MRMesh links `libdeflate::libdeflate_shared` via `find_package(libdeflate CONFIG)` with a `HINTS` path at the thirdparty install prefix.
Test plan
Not in this PR
🤖 Generated with Claude Code