Fix numba caching in multiprocess contexts#2079
Merged
ricardoV94 merged 8 commits intopymc-devs:mainfrom Apr 27, 2026
Merged
Conversation
Numba's per-process FunctionIdentity._unique_ids counter is inherited verbatim across os.fork(), so sibling forks allocate identical uid values to same-qualname functions they fresh-compile next. Their .nbc files then contain linkonce_odr weak symbols with byte-identical mangled names but different bodies; a later process that loads both hits LLVM's weak-merge and dispatches into the wrong body, segfaulting. The fix re-seeds FunctionIdentity._unique_ids to a random 48-bit offset the first time our cache locator is consulted in each process, and again after every fork (via os.register_at_fork). This covers fork, forkserver, and spawn uniformly. Caching still works because .nbc files store the mangled name in their serialized FunctionDescriptor and loading reads the baked name rather than re-deriving it from the current counter. See numba/numba#10486
The previous fix re-seeded FunctionIdentity._unique_ids to a random 48-bit offset on first use and after every fork. Sequential counting from distinct offsets still leaves a nonzero collision probability if the ranges overlap, and the fork hook adds state to maintain. Replace the counter with an iterator that yields uuid.uuid4().int on every next() call. Numba only consumes _unique_ids via next(cls._unique_ids) in FunctionIdentity.from_function, so any iterator of unique ints is a drop-in substitute. uuid.uuid4 reads fresh entropy from os.urandom on every call, so it is intrinsically fork-safe: no os.register_at_fork hook, no re-seed flag, and collision probability is 128-bit negligible. See numba/numba#10486
Pin numba in pyproject.toml (main dep and numba extra) so CI does not silently consume releases that change internals we depend on. Add a weekly GitHub Actions workflow running scripts/bump_numba_upper_bound.py: it rewrites the pin to the canonical "numba>=X.Y,<=A.B.C" form whenever PyPI has a release newer than the cap, or the pin shape has drifted (< cap, no upper bound), and opens a PR. The Tests workflow validates each bump; merging is gated on CI and human review.
ricardoV94
force-pushed
the
cherry_pick_v3
branch
from
1e0084d to
42cdda3
Compare
ricardoV94
force-pushed
the
cherry_pick_v3
branch
from
42cdda3 to
2b8451f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry picked from v3 so we can do one last v2 release