[pipeline] 1/4: Add pipeline executor and placement config types by mthrok · Pull Request #1584 · facebookresearch/spdl

mthrok · 2026-07-06T14:41:40Z

This is part 1/4 of a series (plus a follow-up) adding the .to() region API to the SPDL PipelineBuilder — a declarative way to designate where a run of pipeline stages executes: the main process, a subprocess pool, or a subinterpreter pool. This diff carries the overview for the whole series; the later diffs point back here for the rationale.

Running stages off the main process previously meant attaching a shared ProcessPoolExecutor instance to consecutive pipe() calls and passing fuse_subprocess_stages=True to build(). The engine fused adjacent stages that shared the same executor object into one subprocess pipeline, so the value handed from one stage to the next stayed in the worker instead of being pickled back to the main process between stages. That worked but had real limitations: it was implicit (it keyed off executor object identity, which is easy to get wrong and cannot be expressed as static config), it required a live Executor (so a pipeline could not be fully described as serializable data), it could not pull aggregate/disaggregate/path_variants stages into a region (a fusion run was bounded at those stages), and it had no subinterpreter support.

The .to() API makes the region explicit and declarative. to(ProcessPoolExecutorConfig(...)) or to(InterpreterPoolExecutorConfig(...)) opens a region; every stage after it runs in that worker pool until to(MAIN_PROCESS) closes the region. The target is a serializable spec rather than a live executor, so a pipeline stays expressible as static config; a region may contain aggregate/disaggregate/path_variants; the worker-pool configuration has a single home; and subinterpreter pools (Python 3.14+) are supported alongside subprocess pools. The value passed between stages inside a region never leaves the worker and need not be picklable — only the region's inputs and outputs cross the boundary.

1/4 — this PR ([pipeline] 1/4: Add pipeline executor and placement config types #1584): the config-layer building blocks (the ExecutorConfig spec hierarchy — ProcessPoolExecutorConfig and InterpreterPoolExecutorConfig — plus MAIN_PROCESS and the PlacementConfig region marker), and widening PipelineConfig.pipes to carry region markers. Additive and inert.
2/4 — [pipeline] 2/4: Fuse placement-marked regions into worker-pool pipelines #1585: the engine pass _fuse_marked_regions that consumes the markers, replacing each maximal span of stages under a region target with one stage that runs the span as a nested pipeline in a worker pool. Dormant until markers exist.
3/4 — [pipeline] 3/4: Subinterpreter worker-pool backend for placement regions #1586: the subinterpreter worker-pool backend, behind a _PoolBackend seam, so a region can run in subinterpreters as well as subprocesses.
4/4 — [pipeline] 4/4: Add PipelineBuilder region-placement method and validation #1587: the public surface (PipelineBuilder.to() and its validation), which makes the feature usable end to end.
follow-up — [pipeline] Remove fuse_subprocess_stages #1588 (BC-breaking): removes the superseded fuse_subprocess_stages executor-identity fusion path now that .to() provides the same capability with an explicit, statically-configurable surface.

Adds the config-layer types, additive only — no behavior change yet:

ExecutorConfig: the base spec for a worker-pool executor — a serializable description of a pool (worker count, initializer, initargs) that the pipeline later materializes into a live executor. Kept as a base so more executor targets (e.g. a thread pool) can be added without touching the placement machinery, and so the specs can be reused wherever a pool is configured.
ProcessPoolExecutorConfig / InterpreterPoolExecutorConfig: the two concrete ExecutorConfig subclasses, for a subprocess or subinterpreter worker-pool region. The process variant adds mp_context; the subinterpreter variant adds nothing (no new process is started). They describe the pool without holding a live Executor.
MAIN_PROCESS: a sentinel target that closes a region and brings subsequent stages back to the main process. It reprs as MAIN_PROCESS and, unlike None, is unambiguous and serializes cleanly.
PlacementConfig: a region-marker node carried in PipelineConfig.pipes; stages after a marker (until the next one) run on its target (an ExecutorConfig or MAIN_PROCESS).

PipelineConfig.pipes now accepts PlacementConfig. Nothing produces these markers yet (PipelineBuilder.to() lands in 4/4), so this is inert; the fusion pass that consumes them lands in 2/4.

Note: a region's worker thread count and stats interval are pipeline/build settings, not properties of a placement, so they are not fields of these specs — the engine derives the region's thread count from the concurrency of the stages it fuses and inherits the stats interval from build_pipeline.

meta-codesync · 2026-07-06T14:46:05Z

This pull request has been imported. If you are a Meta employee, you can view this in D110766097. (Because this pull request was imported automatically, there will not be any future comments.)

Follow-up to the `.to()` region API (1/4–4/4; see #1584 for the overall design and rationale). Removes the executor-identity fusion path now that `.to()` regions provide the same capability with an explicit, statically-configurable surface. Both removed features were added in the still-unreleased 0.6.0 cycle, so no deprecation shim is warranted. Removed: - The `fuse_subprocess_stages` keyword from `PipelineBuilder.build`, `build_pipeline`, and `run_pipeline_in_subprocess`. - The executor-identity fusion machinery in `_fuse.py` (`_find_fusable_runs`, `_scan_run`, `_FusableRun`, `_fusable_*`, `_fuse_subprocess_stages`, `_pool_params`, the identity `_build_fused_stage`) and the async-op-as-fusion-tag pass (`_strip_async_executor_tags`). The marker path (`_fuse_marked_regions` and friends) is kept. Reverted (the async-op executor relaxation from #1582): an async op may no longer be given an `executor`. `PipeConfig.__post_init__` again rejects any executor on an async op, and `convert_to_async` asserts it is `None`. In a `.to()` region an async op is placed by the marker and needs no per-stage executor tag, so the relaxation is obsolete. `run_pipeline_in_subprocess` now fuses `.to()` regions in the main process (via `_fuse_marked_regions`), preserving the main-ownership of region worker pools that the old flag provided. Docs: the parallelism guide's "Multi-processing (fused)" section is rewritten as "Multi-processing (region)" using `.to()`.

This is part 1/4 of a series (plus a follow-up) adding the `.to()` region API to the SPDL `PipelineBuilder` — a declarative way to designate where a run of pipeline stages executes: the main process, a subprocess pool, or a subinterpreter pool. This diff carries the overview for the whole series; the later diffs point back here for the rationale. Running stages off the main process previously meant attaching a shared `ProcessPoolExecutor` instance to consecutive `pipe()` calls and passing `fuse_subprocess_stages=True` to `build()`. The engine fused adjacent stages that shared the same executor object into one subprocess pipeline, so the value handed from one stage to the next stayed in the worker instead of being pickled back to the main process between stages. That worked but had real limitations: it was implicit (it keyed off executor object identity, which is easy to get wrong and cannot be expressed as static config), it required a live `Executor` (so a pipeline could not be fully described as serializable data), it could not pull `aggregate`/`disaggregate`/`path_variants` stages into a region (a fusion run was bounded at those stages), and it had no subinterpreter support. The `.to()` API makes the region explicit and declarative. `to(ProcessPoolExecutorConfig(...))` or `to(InterpreterPoolExecutorConfig(...))` opens a region; every stage after it runs in that worker pool until `to(MAIN_PROCESS)` closes the region. The target is a serializable spec rather than a live executor, so a pipeline stays expressible as static config; a region may contain `aggregate`/`disaggregate`/`path_variants`; the worker-pool configuration has a single home; and subinterpreter pools (Python 3.14+) are supported alongside subprocess pools. The value passed between stages inside a region never leaves the worker and need not be picklable — only the region's inputs and outputs cross the boundary. - 1/4 — this PR (#1584): the config-layer building blocks (the `ExecutorConfig` spec hierarchy — `ProcessPoolExecutorConfig` and `InterpreterPoolExecutorConfig` — plus `MAIN_PROCESS` and the `PlacementConfig` region marker), and widening `PipelineConfig.pipes` to carry region markers. Additive and inert. - 2/4 — #1585: the engine pass `_fuse_marked_regions` that consumes the markers, replacing each maximal span of stages under a region target with one stage that runs the span as a nested pipeline in a worker pool. Dormant until markers exist. - 3/4 — #1586: the subinterpreter worker-pool backend, behind a `_PoolBackend` seam, so a region can run in subinterpreters as well as subprocesses. - 4/4 — #1587: the public surface (`PipelineBuilder.to()` and its validation), which makes the feature usable end to end. - follow-up — #1588 (BC-breaking): removes the superseded `fuse_subprocess_stages` executor-identity fusion path now that `.to()` provides the same capability with an explicit, statically-configurable surface. Adds the config-layer types, additive only — no behavior change yet: - `ExecutorConfig`: the base spec for a worker-pool executor — a serializable description of a pool (worker count, initializer, initargs) that the pipeline later materializes into a live executor. Kept as a base so more executor targets (e.g. a thread pool) can be added without touching the placement machinery, and so the specs can be reused wherever a pool is configured. - `ProcessPoolExecutorConfig` / `InterpreterPoolExecutorConfig`: the two concrete `ExecutorConfig` subclasses, for a subprocess or subinterpreter worker-pool region. The process variant adds `mp_context`; the subinterpreter variant adds nothing (no new process is started). They describe the pool without holding a live `Executor`. - `MAIN_PROCESS`: a sentinel target that closes a region and brings subsequent stages back to the main process. It reprs as `MAIN_PROCESS` and, unlike `None`, is unambiguous and serializes cleanly. - `PlacementConfig`: a region-marker node carried in `PipelineConfig.pipes`; stages after a marker (until the next one) run on its target (an `ExecutorConfig` or `MAIN_PROCESS`). `PipelineConfig.pipes` now accepts `PlacementConfig`. Nothing produces these markers yet (`PipelineBuilder.to()` lands in 4/4), so this is inert; the fusion pass that consumes them lands in 2/4. Note: a region's worker thread count and stats interval are pipeline/build settings, not properties of a placement, so they are not fields of these specs — the engine derives the region's thread count from the concurrency of the stages it fuses and inherits the stats interval from `build_pipeline`.

Part 2/4 of the `.to()` region API; see #1584 for the overall design and rationale. Behind-the-scenes engine change — backward compatible and dormant until region markers exist, so it lands before the public `.to()` surface (4/4) without altering any current behavior. Adds `_fuse_marked_regions`: it walks `PipelineConfig.pipes`, tracks the current execution target set by `PlacementConfig` markers (a pipeline starts on the main process), and replaces each maximal span of stages under a `ProcessPoolExecutorConfig` target with one stage that runs the span as a nested pipeline in a worker pool — eliminating inter-stage IPC within the region. Crucially, `aggregate`/`disaggregate`/`path_variants` stages inside a region are absorbed into it, which the existing executor-identity fusion cannot do (it bounds a run at those stages). Pool parameters come from the serializable spec rather than a live executor; the region's worker thread count is derived from the concurrency of the stages it fuses, and the stats interval is inherited from `build_pipeline`. `_build_fused_stage` is refactored to share a core with the new spec-driven builder. `build_pipeline` now runs `_fuse_marked_regions` unconditionally; it is a no-op when the config has no markers, so the existing `fuse_subprocess_stages` path is untouched. Subinterpreter regions raise `NotImplementedError` pending the subinterpreter worker-pool backend (3/4).

Part 3/4 of the `.to()` region API; see #1584 for the overall design. Adds the subinterpreter worker-pool backend so a region can run in subinterpreters, not just subprocesses. Still no public surface — dormant until markers exist. - Factors the process-specific bits of `_SubprocessPipelinePool` behind a small `_PoolBackend` seam (`make_queue`, `spawn`, `try_put_shutdown`, `close_queue`) with two implementations: `_ProcessBackend` (existing `multiprocessing` behavior, unchanged) and `_InterpreterBackend` (`concurrent.interpreters`, Python 3.14+). The streaming protocol and worker body (`_pipeline_worker_loop`) are backend-agnostic and unchanged; the worker is spawned via `interp.call_in_thread` and reaped by joining its thread (subinterpreters can't be force-killed, so teardown relies on the broadcast `_POOL_SHUTDOWN` marker the worker loop already honors). - `_fuse.py` now selects the backend by region-spec type: `ProcessPoolExecutorConfig` → process backend (with the fork+threads warning); `InterpreterPoolExecutorConfig` → subinterpreter backend, or a clear `RuntimeError` on Python < 3.14. Replaces the previous `NotImplementedError`. - `interpreters.Queue` transports items by pickling like `mp.Queue`, so no queue-protocol changes were needed and unpicklable intermediates stay in the worker.

…ation Part 4/4 of the `.to()` region API; see #1584 for the overall design and rationale. The public surface — the engine (#1585) and backend (#1586) already support regions end-to-end, so this makes the feature usable. - Adds `PipelineBuilder.to(target)`: appends a `PlacementConfig` marker designating where the subsequent stages run. `target` is a `ProcessPoolExecutorConfig`, `InterpreterPoolExecutorConfig`, or `MAIN_PROCESS`. The builder stays a stateless append-only list (no region state). A live `Executor` is rejected with a `TypeError` pointing at `pipe(executor=...)`, keeping the pipeline expressible as static config. - Adds stateless validation in `get_config()` (so it also covers `build()` and static configs): rejects a region left open before the sink, a stage with `output_order="input"` inside a region (order cannot be preserved across independent workers), and a subinterpreter region on Python < 3.14. - `fuse_subprocess_stages` is left in place; the two paths coexist until the follow-up removal.