Skip to content

feat: adopt mpz #403 (bounded map concurrency) + rand 0.10 - #1160

Draft
heeckhau wants to merge 7 commits into
mainfrom
feat/mpz-403-integration
Draft

feat: adopt mpz #403 (bounded map concurrency) + rand 0.10#1160
heeckhau wants to merge 7 commits into
mainfrom
feat/mpz-403-integration

Conversation

@heeckhau

@heeckhau heeckhau commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Draft / integration gate. Validates ethereum/mpz#403 and tlsnotary/tlsn-utils#107 against tlsn end-to-end before those PRs merge. Pinned to two unmerged upstream branch SHAs (see below) — do not merge until both land and the pins move to tagged revs.

Why

TooManyStreams in production (verifier at max_sent=8192, max_recv=70000) was caused by mpz_common::Context::map opening one mux stream per work item, so concurrent streams scaled with transcript size and overran the mux default of 512. #1159 only raised the cap — a band-aid. mpz#403 fixes the root cause: map is bounded to a constant concurrency (default 32) via buffered, so at most N streams are open at once regardless of work.

Backport measurement (mux at default 512, no cap bump): peak concurrent streams at the production config dropped 555 → ~133; the smaller test_mpc config dropped 285 → 94.

What

mpz#403 — bounded executor + rand 0.10

  • session.rs: migrate mpz_common::Executor/ExecutorBuilder → new Session/SessionBuilder + ThreadPool API.
  • Bump workspace rand 0.9 → 0.10 to match mpz, with the API fallout: fill → fill_bytes/RngExt; thread_rng → rng; .gen() → .random(); RngCore no longer re-exported.

tlsn-utils#107 — rand06-compat 0.6↔0.10 + mux on rand 0.10

  • Bump tlsn-utils deps (mux/futures-plex/spansy/websocket-relay) to Simplified 2PC point addition #107.
  • Use the updated rand06-compat (now bridges rand_core 0.6↔0.10) for p256/k256 key generation — no local adapter needed.
  • Drop the unused rand06-compat@0.1 and, since tlsn-mux now uses rand 0.10, the getrandom_03 wasm dependency. getrandom in the wasm closure is down from three versions to two (0.2 EC stack, 0.4 rand 0.10).

Test status

  • ✅ Full workspace unit tests.
  • ✅ tlsn ignored integration tests (test_mpc, test_proxy) — full MPC protocol end-to-end on the new executor.
  • ⏳ wasm (crates/wasm/build.sh) — in progress; getrandom consolidation should unblock it. Native-only verified so far.

⚠️ Temporary pins (must change before merge)

Upstream Pinned SHA
ethereum/mpz#403 3acfac3
tlsnotary/tlsn-utils#107 6060113

Re-pin both to tagged revs once they merge, then re-run CI.

🔍 Review notes

  • Crypto-sensitive: rand 0.9 → 0.10 in an MPC protocol — RNG semantics warrant scrutiny.
  • OsRng → ThreadRng: rand 0.10 dropped rngs::OsRng (now SysRng); two EphemeralSecret::random(&mut OsRng) sites in tls-client now use rng().compat() (ThreadRng CSPRNG). Confirm acceptable for ephemeral keys.

TODO before un-drafting

  • mpz#403 merged; re-pin mpz-* to a tagged rev
  • tlsn-utils#107 merged; re-pin tlsn-utils deps to a tagged rev
  • wasm build verified (build.sh)
  • add a regression test at the production config (max_recv=70000)
  • (future) consolidate to a single getrandom once the EC stack (k256/p256) moves to rand_core 0.9+

Refs: ethereum/mpz#403, tlsnotary/tlsn-utils#107, #1159

heeckhau added 2 commits June 2, 2026 11:19
@heeckhau
feat: adopt mpz #403 (bounded map concurrency) + rand 0.10
e2f5c19 
Integrates the mpz executor refactor (ethereum/mpz#403), which
bounds `Context::map` concurrency to a constant (default 32) via
`buffered`, so the number of concurrently open mux streams no longer
scales with protocol work. This fixes `TooManyStreams` at the root
rather than by raising the mux cap (cf. tlsn#1159). Measured peak streams
at the production config (max_recv=70000) drop 555 -> ~133, under the
mux default of 512.

Changes:
- session.rs: migrate from `mpz_common::Executor`/`ExecutorBuilder` to
  the new `Session`/`SessionBuilder` + `ThreadPool` API.
- Bump workspace rand/rand_chacha/rand_core 0.9 -> 0.10 to match mpz;
  work through the API fallout (`fill`->`fill_bytes`/`RngExt`,
  `thread_rng`->`rng`, `gen`->`random`, `RngCore` no longer re-exported).
- Replace `rand06-compat` (bridges rand_core 0.6<->0.9 only) with a local
  `Compat` adapter bridging rand_core 0.10 -> 0.6 for p256/k256 key gen.

TEMPORARY (revert before merge):
- mpz-* are pinned to the #403 branch SHA 3acfac3, not a release. Re-pin
  to a tagged mpz rev once #403 merges to dev.

REVIEW NOTES:
- crypto-sensitive: rand 0.10 upgrade in an MPC protocol; the `Compat`
  adapter and RNG semantics need scrutiny.
- two `EphemeralSecret::random(&mut OsRng)` sites now use
  `Compat(rand::rng())` (ThreadRng, a CSPRNG) since rand 0.10 dropped
  `rngs::OsRng` in favor of `SysRng`; confirm this substitution is
  acceptable for ephemeral key generation.
- the `Compat` adapter is duplicated in key-exchange, tls-client, and
  attestation; consider a shared helper crate.
@heeckhau
fix(wasm): enable getrandom 0.4 wasm_js feature for rand 0.10
1077ecf 
rand 0.10 (via mpz) pulls getrandom 0.4, whose wasm backend is gated
behind the wasm_js crate feature rather than the getrandom_backend cfg
that .cargo/config.toml sets for getrandom 0.3. Add a renamed
getrandom_04 dependency on the wasm target to enable it, mirroring the
existing getrandom_03 entry.
@heeckhau
heeckhau force-pushed the feat/mpz-403-integration branch from 763d21f to 1077ecf Compare June 2, 2026 10:19
@heeckhau
chore: drop unused rand06-compat dependency
b0a16f3 
All rand06-compat usages were replaced by the local Compat adapter
(rand_core 0.10 -> 0.6 bridge), so the crate is no longer referenced.
Removing it also drops one consumer of rand_core 0.9.
@sinui0

sinui0 commented Jun 2, 2026

Copy link
Copy Markdown
Member

a note to make sure to use the global thread pool. If one is not present then configure and install it as the global. This will close #1081

@heeckhau

heeckhau commented Jun 2, 2026

Copy link
Copy Markdown
Member Author

@sinui0 WIP at #1160

heeckhau added 4 commits June 2, 2026 22:50
@heeckhau
refactor: use rand06-compat from tlsn-utils #107 instead of local ada…
cf720c8 
…pters

tlsn-utils#107 updates rand06-compat to bridge rand_core 0.6 <-> 0.10
(it previously only did 0.6 <-> 0.9) and bumps tlsn-mux to rand 0.10.

- Bump tlsn-utils deps (mux/futures-plex/spansy/websocket-relay) to the
  #107 head SHA 6060113.
- Delete the three local Compat adapters (key-exchange, tls-client,
  attestation) and the tls-client rand_compat module; restore the
  upstream .compat() / Rand0_6CompatExt usage.
- Drop the getrandom_03 wasm dependency: tlsn-mux now uses rand 0.10, so
  getrandom 0.3 is no longer pulled into the wasm crate. getrandom in the
  wasm closure is down from three versions to two (0.2 EC stack, 0.4 rand).

TEMPORARY (revert before merge): tlsn-utils deps pinned to the #107
branch SHA; re-pin to a tagged rev once #107 merges.

All workspace unit tests and the tlsn MPC integration tests pass.
@heeckhau
Missing lock file
5df3d3d
@heeckhau
test: assert mpz#403 bounds concurrent streams in Context::map
23d9b99 
Adds an integration test that drives Context::map over 1000 items through
a custom Mux counting concurrent open streams, and asserts the peak stays
within DEFAULT_CONCURRENCY_LIMIT (+1 for the parent context channel).

Measured peak is 33 (= 1 baseline + 32 limit); an unbounded map would
peak at ~1001. The test fails if mpz is ever bumped to a version whose
map opens one stream per item again, guarding the fix tlsn relies on to
avoid TooManyStreams during preprocessing.
@heeckhau
chore: point tlsn-utils at e99c00d (dev + #111 backpressure incl. los…
a7cddd2 
…t-close fix)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@heeckhau @sinui0