Skip to content

fix: use per-extension subdirectory in build_temp to prevent parallel build race (#5196)#5202

Open
bibekmhj wants to merge 1 commit intopypa:mainfrom
bibekmhj:fix/parallel-build-race-condition-5196
Open

fix: use per-extension subdirectory in build_temp to prevent parallel build race (#5196)#5202
bibekmhj wants to merge 1 commit intopypa:mainfrom
bibekmhj:fix/parallel-build-race-condition-5196

Conversation

@bibekmhj
Copy link
Copy Markdown

Summary

Fixes #5196

Root cause

All extensions share the same build_temp directory when compiled
in parallel with -j. The compiler.compile() call uses
output_dir=self.build_temp for every extension, so .o files
with the same name overwrite each other.

Fix

Derive a per-extension output_dir from ext.name:

ext_build_temp = os.path.join(self.build_temp, *ext.name.split('.'))

pkg.ext1build/temp.../pkg/ext1/
pkg.ext2build/temp.../pkg/ext2/

Testing

Added unit test verifying the two extensions receive different
output_dir values.

… build race

When building multiple extensions in parallel with -j, all extensions
shared the same build_temp directory. Object files with the same name
would overwrite each other, causing random build failures.

Fixed: derive a per-extension output_dir from ext.name.split('.').
e.g. pkg.ext1 -> build/temp.../pkg/ext1/
     pkg.ext2 -> build/temp.../pkg/ext2/

Fixes pypa#5196
@Avasam
Copy link
Copy Markdown
Contributor

Avasam commented Apr 13, 2026

Please note that https://setuptools.pypa.io/en/latest/development/developer-guide.html#code-conventions-and-other-practices

Finally, the setuptools/_distutils directory should not be modified directly when contributing to the setuptools project. Instead, this directory is maintained as a separated project in https://github.com/pypa/distutils, and periodically merged into setuptools.

and #5196 (comment)

One thing that requires attention though is that changes in the setuptools/_distutils directory need to be addressed in the pypa/distutils repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Race condition due to reuse of build folder when building extensions in parallel

2 participants