Skip to content

Further tweak CacheDir for races#4844

Open
mwichmann wants to merge 1 commit intoSCons:masterfrom
mwichmann:fix/cachedir-race-again
Open

Further tweak CacheDir for races#4844
mwichmann wants to merge 1 commit intoSCons:masterfrom
mwichmann:fix/cachedir-race-again

Conversation

@mwichmann
Copy link
Copy Markdown
Collaborator

Some CacheDir race conditions appear to still exist in the field. The existing mechanism tries to create a random-enough temporary filename to use for the copy of the object to be cached, before doing an atomic rename to the final filename in the cache. The temporary filename uses a uuid suffix which has a miniscule chance of not being unique across SCons invocations, but which is unchanging within a single invocation. The new proposal is to use Python's tempfile module to create a unqiue temporary directory for each cache write, copy the file there, then do the rename. One could further imagine wrapping critical code sections in filelock calls but that's not done in this change (FileLock is currently to protect initial CacheDir creation operations as that had been a source of conflicts in a multi-threaded build, but not for individual cache writes or reads).

There is no test: I spent a fair bit of time with AI assistance trying to create a reliable reproducer without success - through ever more contrived scenarios it still wouldn't fail. So while this may help, it's still a bit of "flying blind".

Contributor Checklist:

  • I have created a new test or updated the unit tests to cover the new/changed functionality.
  • I have updated CHANGES.txt and RELEASE.txt (and read the README.rst).
  • I have updated the appropriate documentation

Some race conditions appear to still exist in the field. The existing
mechanism tries to create a random-enough temporary filename to use for
the copye of the object to be cached, before doing an atomic rename to
the final filename in the cache. The temporary filename uses a uuid
suffix which has a miniscule chance of not being unique across SCons
invocations, but which is unchanging within a single incovation.  The new
proposal is to use Python's tempfile module to create a unqiue temporary
directory for each cache write, copy the file there, then do the rename.
One could further imagine wrapping critical code sections in filelock
calls but that's not done in this change (FileLock is currently used
for initial cachedir creation operations but not for individual cache
writes or reads).

Signed-off-by: Mats Wichmann <mats@linux.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant