framework: add parallelism overcommit and annotate heavy examples#1533
Merged
framework: add parallelism overcommit and annotate heavy examples#1533
Conversation
Add a `parallelism_overcommit` setting that controls how much extra parallelism build tools get beyond the Bazel scheduler reservation. A target with cpu=4 and overcommit=2 (the default) gets `-j6` in its env vars while Bazel still schedules it as needing 4 cores. This mirrors ninja's own ncpus+2 default, which slightly oversubscribes to hide I/O latency. It also helps configure_make targets whose configure phase is serial: the make phase gets more parallelism without inflating what the scheduler sees. The setting is an `int_flag` at `//foreign_cc/settings:parallelism_overcommit`, following the same pattern as the existing size settings. The "serial" size is exempt via `fixed_cpu` — it always enforces exactly `-j1`. Also refactors `get_resource_set()` to return a struct with an `allow_cpu_overcommit` field, and rewrites the `resource_size` attribute doc for clarity.
Give Bazel scheduling information for the heaviest build targets so it can overlap them more effectively. On an 8-core machine this reduces uncached build time by ~25% (506s → 378s) and critical path by ~27% (488s → 355s). Only targets where the build phase (make/ninja) dominates are annotated — meson/ninja targets (mesa, glib) are left at default because setting NINJA_JOBS caps ninja below its auto-detected parallelism. Annotations: - enormous (cpu=4): openssl (both variants), python3, subversion, cmake_tool (the cmake bootstrap is 86% of the Main job critical path) - large (cpu=3): gn, curl, log4cxx The .bazelrc.common settings rescale enormous/large to fit 8 cores by default.
novas0x2a
force-pushed
the
resource-sets-rework
branch
from
b4ef824 to
cf3c784
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a
parallelism_overcommitsetting and useresource_sizeannotations onthe heaviest build targets to give Bazel better scheduling information. On an
8-core machine this reduces uncached Examples build time by ~25% (506s → 378s)
and critical path by ~27% (488s → 355s).
The overcommit setting (
//foreign_cc/settings:parallelism_overcommit,default +2) lets build tools use slightly more parallelism than the scheduler
reservation, mirroring ninja's own ncpus+2 convention. This hides I/O latency
and helps configure_make targets whose configure phase is serial. The "serial"
size is exempt — it always enforces
-j1.Meson/ninja targets (mesa, glib) are deliberately left unannotated: setting
NINJA_JOBScaps ninja below its auto-detected parallelism, making individualtargets ~2x slower and hurting the overall build.
Annotations:
(cmake bootstrap is 86% of the Main job critical path at 235s)
The
.bazelrc.commonsettings rescale enormous/large to fit 8 cores bydefault.