Skip to content

feat(pkg): solve all requested platforms together - #15982

Draft
Alizter wants to merge 5 commits into
ocaml:mainfrom
Alizter:push-kzszzntwnrxt
Draft

feat(pkg): solve all requested platforms together#15982
Alizter wants to merge 5 commits into
ocaml:mainfrom
Alizter:push-kzszzntwnrxt

Conversation

@Alizter

@Alizter Alizter commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Build one SAT universe containing every requested platform instead of solving each platform independently.
  • Make roles, availability, conflict classes, and result extraction platform-aware.
  • Constrain every platform role for a package to select the same version, choosing an older common version when necessary and failing when no common version exists.
  • Resolve each platform's package formulas against only the packages selected on that platform, preserving platform-specific dependency alternatives.
  • Generate conditional lock-file data directly from the joint result and remove obsolete result-merging APIs.
  • Credit: the cross-platform version-equality SAT encoding was proposed by @art-w in portable lock directories should pick the same version for cross-platform packages #13647.

Pre-feature tests

This change merges three independent draft test branches before applying the joint-solver implementation:

Review fixes

  • Compute post-solve reachability separately for every platform before taking the union, so a local package's macOS alternative is not dropped after its Linux alternative is visited first.
  • Keep versions rejected on every platform out of manifest loading, while retaining platform-local candidates needed by another platform.
  • Report cross-platform version conflicts with the selected version and platform instead of an opaque SAT clause.
  • Rename and rewrite the no-common-version regression so its name and prose match the enforced invariant.
  • Remove the redundant platform-version extra-files regression at the commit where joint version equality makes it obsolete.

Checks

  • dune runtest test/blackbox-tests/test-cases/pkg/portable-lockdirs
  • CI=true dune build @fmt @check

Closes #13647

@Alizter Alizter added the package management Dune's package management — `(pkg)` stanza, lockdirs, `dune pkg` commands label Aug 17, 2026
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch 3 times, most recently from 6783d45 to bd35eab Compare August 17, 2026 11:39
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch from bd35eab to 33fe18e Compare August 17, 2026 11:50
@Alizter
Alizter marked this pull request as ready for review August 17, 2026 11:53
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch from 33fe18e to fa8b474 Compare August 17, 2026 14:08
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch from fa8b474 to ff35b1a Compare August 17, 2026 16:32
Alizter added a commit that referenced this pull request Aug 17, 2026
## Summary

- Add a trace-based regression for a lock directory with exactly one
requested platform.
- Assert the exact SAT problem size for a fixed one-package input: two
variables and two clauses.
- Guard against adding cross-platform consistency encoding to the
single-platform path.

This is an independent test-only performance baseline for the
joint-solver work in #15982.

## Tests

- `nix develop -c dune runtest
test/blackbox-tests/test-cases/pkg/portable-lockdirs/portable-lockdirs-single-platform-sat-size.t`
- `CI=true nix develop -c dune build @fmt @check`
Alizter added a commit that referenced this pull request Aug 17, 2026
## Summary

- Add a portable lock-directory regression with distinct extra files
attached to two versions of one package.
- Constrain the solve to the common selected version and assert that its
file is copied into the lock directory.
- Assert that files belonging only to the rejected version are omitted.

This is an independent test-only baseline for per-platform lock
extraction in #15982.

## Tests

- `nix develop -c dune runtest
test/blackbox-tests/test-cases/pkg/portable-lockdirs/portable-lockdirs-common-version-extra-files.t`
- `CI=true nix develop -c dune build @fmt @check`
Alizter added a commit that referenced this pull request Aug 17, 2026
## Summary

- Add a portable lock-directory regression for an unfiltered dependency
alternative whose implementations are available on different platforms.
- Assert that locking preserves the implementation selected for each
platform.
- Build the resulting lock directory on Linux and macOS to cover
consumption as well as the lock summary.

This snapshots the existing per-platform selection invariant before the
joint-solver change in #15982.

## Tests

- `nix develop -c dune runtest
test/blackbox-tests/test-cases/pkg/portable-lockdirs/portable-lockdirs-platform-alternative-selection.t`
- `CI=true nix develop -c dune build @fmt @check`
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch from ff35b1a to bd76ef4 Compare August 17, 2026 18:07
@Alizter
Alizter marked this pull request as draft August 17, 2026 18:12
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch from bd76ef4 to d9132ec Compare August 17, 2026 18:44
Alizter added a commit that referenced this pull request Aug 17, 2026
## Summary

- Add a regression documenting that existing language versions currently
allow a portable lock directory to choose different versions of one
package on different platforms.
- Use mutually exclusive Linux and macOS availability filters to make
that behavior explicit.
- Establish the compatibility baseline changed by the joint solver.

This test intentionally documents legacy behavior; #15982 updates its
expectation to the intended all-platform failure when no common version
exists.

## Tests

- `nix develop -c dune runtest
test/blackbox-tests/test-cases/pkg/portable-lockdirs/portable-lockdirs-language-version-compatibility.t`
- `CI=true nix develop -c dune build @fmt @check`
Alizter added a commit that referenced this pull request Aug 17, 2026
## Summary

- Add a portable lock-directory regression for a local package whose
dependency and conflict formulas contain platform filters.
- Request Linux explicitly and assert that `foo.2` remains selectable
there.
- Guard against evaluating local-package formulas in a platform-less
environment before multi-platform solving.

This is an independent test-only baseline for the joint-solver work in
#15982.

## Tests

- `nix develop -c dune runtest
test/blackbox-tests/test-cases/pkg/portable-lockdirs/portable-lockdirs-local-platform-constraints.t`
- `CI=true nix develop -c dune build @fmt @check`
Alizter added a commit that referenced this pull request Aug 17, 2026
## Summary

- Add a portable lock-directory regression for a package that exists in
the repository but is unavailable on Linux.
- Assert that the solver reports `Availability condition not satisfied`
for the requested Linux platform.
- Guard against reconstructing rejection reasons in a platform-less
environment.

This is an independent test-only baseline for the joint-solver work in
#15982.

## Tests

- `nix develop -c dune runtest
test/blackbox-tests/test-cases/pkg/portable-lockdirs/portable-lockdirs-platform-rejection-reason.t`
- `CI=true nix develop -c dune build @fmt @check`
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch 4 times, most recently from 5a6beb1 to 4530788 Compare August 18, 2026 10:54
@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch 2 times, most recently from bd79350 to b9a87f4 Compare August 18, 2026 12:31
@Alizter

Alizter commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

SAT solver benchmark

I benchmarked the stack against its pre-joint-solver baseline on the 10 real repositories used in the earlier LMDB benchmark: re, cohttp, jsoo, lsp, opam, irmin, mirage, eio, owl, and tezos.

As it currently stands, the joint solver has an approximately 2× slowdown in SAT-solving time. The geometric mean of the per-project new / baseline ratios is 2.013×. All 10 projects were slower; ratios ranged from 1.071× (Irmin) to 2.724× (opam).

Project Baseline mean Joint mean Joint / baseline
re 65.7 ms 165.0 ms 2.511×
cohttp 402.4 ms 569.0 ms 1.414×
jsoo 27.1 ms 55.5 ms 2.047×
lsp 10.2 ms 21.1 ms 2.057×
opam 2.1 ms 5.7 ms 2.724×
irmin 166.7 ms 178.5 ms 1.071×
mirage 7.4 ms 16.2 ms 2.178×
eio 4.7 ms 12.3 ms 2.642×
owl 4.9 ms 11.6 ms 2.364×
tezos 18.3 ms 33.6 ms 1.839×

Method: baseline c58489c, joint-solver head ce68d06f, pinned opam repository and project revisions, 2 warmups plus 10 measured runs per project/revision, with versions interleaved. For each dune pkg lock, I summed the durations of all complete sat/solve trace events. This matters because the baseline normally emits four per-platform events while the joint solver emits one combined event.

The result measures time inside the SAT solver, not end-to-end lock-generation wall time. The reduced solve count is present, but the single combined solve currently costs more than the four baseline solves summed.

@Alizter

Alizter commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up: end-to-end lock-generation time

An important clarification to the SAT-only results above: the joint solver is faster end-to-end, despite spending more time in the timed sat/solve section.

I reran baseline c58489c and joint-solver head ce68d06f on the same pinned inputs using fully source-built Dune binaries and measured complete dune pkg lock wall time. The joint implementation was faster on all 10 projects, with a geometric-mean 1.75× end-to-end speedup:

Project Baseline lock time Joint lock time Speedup
re 0.473 s 0.243 s 1.95×
cohttp 0.947 s 0.546 s 1.73×
jsoo 0.411 s 0.248 s 1.66×
lsp 0.592 s 0.543 s 1.09×
opam 0.347 s 0.161 s 2.16×
irmin 1.116 s 0.717 s 1.56×
mirage 0.411 s 0.207 s 1.99×
eio 0.394 s 0.202 s 1.95×
owl 0.494 s 0.243 s 2.03×
tezos 0.681 s 0.416 s 1.64×

These were interleaved runs with one warmup and five measured observations per project/revision. The SAT regression is therefore real but does not translate into a user-visible lock-generation regression: the reduction in work outside the timed SAT section more than compensates for it.

@Alizter

Alizter commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator Author

In a follow up change, we have the opportunity to share a lot of the traversal work that the combined solves do. This dramatically speeds up the solve times compared the cumulative solves. Therefore I don't think any of the performance differences highlighted here will be a serious issue.

@Alizter
Alizter force-pushed the push-kzszzntwnrxt branch 2 times, most recently from 7046ca3 to 2611e1d Compare August 18, 2026 16:57
Signed-off-by: Ali Caglayan <alizter@gmail.com>
Verify portable locking succeeds when every requested platform can install the dependency.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
Record the duplicate-result merge failure and repeated failure diagnostics before joint solving deduplicates requested platforms.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
Record that separate platform solves select distinct platform-specific avoid-version packages instead of one common alternative.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
Build one SAT universe for every requested platform instead of solving each
platform independently. Enforce one version per package name across platforms
while allowing different package-name alternatives, evaluate availability and
local-package constraints in each platform environment, and minimize distinct
avoid-version package versions.

Keep the single-platform SAT encoding unchanged, preserve each selected
platform package when extracting lock data, and only copy extra files for
selected package versions.

Signed-off-by: Ali Caglayan <alizter@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

package management Dune's package management — `(pkg)` stanza, lockdirs, `dune pkg` commands

Projects

None yet

Development

Successfully merging this pull request may close these issues.

portable lock directories should pick the same version for cross-platform packages

1 participant