From f9b7831a371b0a430cbcebad3d0be9b729a82430 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 30 Jul 2026 11:32:59 -0700 Subject: [PATCH 01/12] =?UTF-8?q?RFC-0003:=20multiverse=20analyses=20?= =?UTF-8?q?=E2=80=94=20in-file=20universes=20and=20artifact@universe=20ref?= =?UTF-8?q?erences?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drafts the RFC for the change discussed in #52, grounded in the astra-multiverse-example prototype (Steegen et al. 2016 Figure 1 reproduction). Also syncs the RFC index status for RFC-0002 (Accepted). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 367 +++++++++++++++++++++++++++++++ rfcs/README.md | 3 +- 2 files changed, 369 insertions(+), 1 deletion(-) create mode 100644 rfcs/0003-multiverse-analyses.md diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md new file mode 100644 index 00000000..ed17bb89 --- /dev/null +++ b/rfcs/0003-multiverse-analyses.md @@ -0,0 +1,367 @@ +--- +rfc: 0003 +title: Multiverse analyses — in-file universes and cross-universe artifact references +status: Draft # Draft | Active | Accepted | Rejected | Superseded +authors: + - Francois Lanusse (@eiffl) +created: 2026-07-30 +tracking-issue: https://github.com/LightconeResearch/astra-spec/issues/52 +superseded-by: +--- + +## Context + +ASTRA can *describe* a multiverse but cannot *analyze* one. The schema records +decisions, options, and per-universe selections — yet the analyses that motivated +calling these things "universes" in the first place (Steegen et al. 2016) are +inexpressible, because no step can consume an artifact realized under a decision +configuration other than its own. Two gaps combine to cause this: + +- **Universes live outside `astra.yaml`.** A universe is a standalone YAML file + in the project's `universes/` directory (e.g. `universes/baseline.yaml`). + Nothing inside `astra.yaml` can name, address, or quantify over a universe. +- **Artifact references are universe-implicit.** Every entry in `Output.inputs` + resolves to an Input or sibling Output *in the current universe*: each + dependency edge silently means "the version of this artifact materialized + under the same decision selections as me." + +The consequence is that a whole class of analyses cannot be declared: + +- averaging or summarizing a metric across the decision space (multiverse + analysis, Steegen et al. 2016), +- a specification-curve figure over the full decision grid (Simonsohn et + al. 2020), +- a robustness comparison between two named configurations (`baseline` vs + `svm_focused`). + +The spec already gestures at the missing machinery: `Output.decisions` +documentation says runners use it to "determine the minimal universe set needed +to materialize the output" — but offers no syntax for an output whose inputs +*span* universes. + +**Prior art.** Multiverse analysis (Steegen, Tuerlinckx, Gelman & Vanpaemel +2016) and specification-curve analysis (Simonsohn, Simmons & Nelson 2020) are +established methodology; tooling such as [Boba](https://github.com/uwdata/boba) +(Liu et al. 2021) demonstrates a DSL for authoring and executing decision +multiverses. On the workflow side, Snakemake's `expand()` is the standard +pattern for fanning a rule's inputs over a parameter grid — the same fan-in +shape this RFC needs at the specification level. + +**Working prototype.** The mechanism proposed here is demonstrated end-to-end by +[`astra-multiverse-example`](https://github.com/anthonyozerov/astra-multiverse-example) +(Anthony Ozerov), an independent reproduction of Figure 1 of Steegen et +al. 2016. Its `astra.yaml` declares the paper's five data-processing decisions; +option constraints (`incompatible_with`) cut the Cartesian product down to 210 +valid universes, and a `when` condition on the Study 1 output reduces *its* +active set to 120. A single `multiverses` entry names the full valid decision +space, six metric outputs are referenced as `@full_multiverse` by a +figure output, and a runner executes all universes and reproduces the paper's +published significance counts. As with RFC-0002, this RFC is grounded in +something that works; its job is to decide what of it belongs in the ASTRA +specification. + +## Proposal + +In plain language: **universes become addressable elements of the analysis +document, sets of universes get names, and an output's inputs may reference an +artifact *at* a universe or *across* a set of universes.** A plain reference +keeps today's meaning — same universe as the consumer — so existing analyses are +untouched. The proposal has three parts. + +### 1. Bring `universes` definitions into `astra.yaml` + +`Analysis` gains an optional **`universes`** slot holding the same `Universe` +objects that today live as standalone files in the `universes/` directory: + +```yaml +universes: + - id: baseline + description: Default configuration using standard practices + decisions: + scaling: standard + model: random_forest + - id: svm_focused + description: Configuration optimized for SVM (requires standard scaling) + decisions: + scaling: standard + model: svm +``` + +The `Universe` class itself is unchanged; what changes is where universes can +live. In-file universes are addressable by the reference grammar below (and by +the RFC-0002 tree-path addressing); the `universes/` directory remains valid for +runner-selected configurations (see *Migration*). + +### 2. Add `multiverses` — named sets of universes + +`Analysis` gains an optional **`multiverses`** slot. A `Multiverse` names a set +of universes so cross-universe steps can reference the set by a stable id and +readers can see, in one place, which decision space a summary quantifies over: + +```yaml +multiverses: + - id: model_robustness + description: Configurations relevant to the model-choice robustness check. + universes: [baseline, svm_focused] + + - id: full_multiverse + description: The valid Cartesian product across all decisions. + universes: "*" +``` + +Membership takes one of two forms: + +- **An enumerated list** of universe ids declared under `universes:`. +- **`"*"`** — every valid point of the decision space: the Cartesian product of + all decisions' options, minus combinations excluded by `requires` / + `incompatible_with` constraints. These grid-point universes are *implicit*: + they need not (and in practice cannot, at 210 of them) be declared + individually. Each receives a **derived id**, formed by joining the selected + option ids in decision-declaration order with hyphens + (e.g. `f5-nmo3-r2-ecl2-ec1`). Option ids are snake_case and may not contain + hyphens, so the derivation is unambiguous and reversible. + +Note the refinement relative to the tracking issue: there, `"*"` was sketched as +"all *declared* universes." The prototype clarified that the useful meaning is +the **full valid decision space** — multiverse analyses quantify over the +choices themselves, not over whichever configurations happen to be named. A set +of declared universes is expressed by enumerating them. + +### 3. Extend the artifact reference grammar with `@` + +Entries in `Output.inputs` may qualify an artifact reference with a universe or +multiverse scope: + +| Reference | Meaning | +|---|---| +| `artifact` | Unchanged: the artifact in the *consumer's own* universe. | +| `artifact@` | The artifact materialized under one declared universe — a **pin**. | +| `artifact@` | The artifact under *each* universe in the set — a **fan-out**, resolving to a collection. | + +Semantics: + +- **Fan-outs respect activation.** `artifact@` resolves to the + artifact's realizations in the universes of the set where the artifact is + *active* — universes excluded by the output's `when` conditions contribute + nothing. (In the prototype, `religiosity_study1_p@full_multiverse` yields 120 + artifacts while its siblings yield 210.) +- **Universe-invariant consumers.** An output whose inputs are all pinned or + fanned-out does not itself vary with the current universe: it is materialized + once per project, not once per universe — the right identity for a + multiverse-level summary or figure. An output that mixes plain and qualified + references remains universe-scoped and additionally pulls in the referenced + cross-universe artifacts. +- **Recipe surface.** In recipe templates, `{inputs.}` for a fan-out + reference expands to the collection of materialized artifact paths; how the + collection is surfaced (space-separated paths, a manifest file, a sidecar) + remains the runner's choice, consistent with the existing recipe contract. +- **Materialization sharing.** Two universes that differ only in decisions *not* + listed on the upstream output's `decisions` share a realization; runners + already compute per-output cache keys this way. A fan-out delivers one entry + per member universe, with shared realizations deduplicated at the cache, not + in the collection (see *Questions*). + +Bare `artifact@*` (option 2 in the tracking issue) is **not** proposed: a +fan-out must name a declared multiverse. This keeps every cross-universe edge +attached to a described, reusable set — one extra stanza in exchange for the +summary's domain being part of the record. + +## Examples + +**A real multiverse analysis** — trimmed from the prototype's reproduction of +Steegen et al. (2016) Figure 1. Five decisions with option constraints define +210 valid universes; each p-value output is computed per universe; the figure +consumes all of them: + +```yaml +decisions: + fertility_assessment: # F1–F5 + ... + next_menstrual_onset: # NMO1–NMO3 + ... + cycle_length_exclusion: + options: + ecl1: { label: ECL1 } + ecl2: + label: ECL2 + incompatible_with: [next_menstrual_onset.nmo2] + ecl3: + label: ECL3 + incompatible_with: [next_menstrual_onset.nmo1] + ... + +outputs: + - id: religiosity_study1_p + type: metric + inputs: [study1_data] + decisions: [fertility_assessment, next_menstrual_onset, + relationship_status, cycle_length_exclusion, + certainty_exclusion] + when: ["~next_menstrual_onset.nmo3"] # NMO3 unavailable in Study 1 + recipe: + command: >- + python src/analyze_universe.py --data {inputs.study1_data} + --outcome religiosity ... --output {output} + + - id: fig1 + type: figure + description: Six-panel histogram of interaction p-values across universes. + inputs: + - religiosity_study1_p@full_multiverse # 120 artifacts + - religiosity_study2_p@full_multiverse # 210 artifacts + - fiscal_attitudes_p@full_multiverse + - social_attitudes_p@full_multiverse + - voting_preference_p@full_multiverse + - donation_preference_p@full_multiverse + recipe: + command: >- + python src/plot_figure1.py --multiverse-results {inputs} + --significance 0.05 --output {output} + +multiverses: + - id: full_multiverse + description: >- + The valid Cartesian product across all five decisions. Option + constraints remove NMO1+ECL3 and NMO2+ECL2, leaving 210 universes. + universes: "*" +``` + +**Named-universe comparison** — the iris example, extended with a robustness +check pinned to two declared configurations: + +```yaml +outputs: + - id: f1_score + type: metric + inputs: [trained_output] + decisions: [scaling, model, test_size, random_seed] + recipe: + command: python src/evaluate.py + + - id: model_comparison + type: metric + description: Baseline vs SVM macro-F1 comparison. + inputs: + - f1_score@baseline + - f1_score@svm_focused + recipe: + command: python src/compare.py + +universes: + - id: baseline + decisions: { scaling: standard, model: random_forest, + test_size: small, random_seed: seed_42 } + - id: svm_focused + decisions: { scaling: standard, model: svm, + test_size: small, random_seed: seed_42 } +``` + +## Implementation implications & migration + +**`astra-spec` (this repo) — schema, datamodel, docs:** + +- `src/astra/schema/analysis.yaml`: add optional `universes` and `multiverses` + slots to `Analysis` (the `universe` schema is already imported); extend the + `Output.inputs` documentation and validation pattern to admit the `@` + qualifier. +- `src/astra/schema/universe.yaml`: add a `Multiverse` class (`id`, + `description`, `universes: list | "*"`). Revisit the `UniverseNode.universe` + doc-string, which currently hard-codes the "sub-analysis's `universes/` + directory." +- Generated artifacts: `just gen-python`, `just gen-doc`; the published JSON + Schema at `astra-spec.org//schema/…` shifts, which is the propagation + point for both SDKs. +- Docs: `specification.md` (the *Universes* section currently states universes + are "stored separately from `astra.yaml`"; add the reference grammar and a + *Multiverses* section), `index.md` at-a-glance example if touched, `cli.md` + validation rules, `README.md`, and the `examples/` projects. A Steegen-style + example project would exercise the full mechanism in-tree. + +**`astra-tools` (Python CLI + SDK):** + +- `astra validate` gains checks: every `@` target resolves to a declared + universe or multiverse id; declared universes select valid options and honor + `requires` / `incompatible_with`; enumerated multiverse members exist; a + pinned reference targets an output that is active in the pinned universe. +- Runner semantics: cross-universe edges change scheduling — one output can now + demand materialization of an upstream artifact under many universes — and the + derived-id scheme for `"*"` grid points must match the spec so artifact paths + are stable across runners. + +**`astra-typescript` (`@astra-spec/sdk`):** regenerate types for the new slots +and the `Multiverse` class; the reference grammar change surfaces wherever the +SDK parses `Output.inputs`. + +**Compatibility / versioning:** + +- Plain references keep their exact current meaning, and both new slots are + optional, so existing analyses validate unchanged: the schema change is + **additive — a minor bump** under the versioning policy. +- The `universes/` directory remains supported as a place for runner-selected + configurations; in-file `universes` are required only when a universe must be + *referenced* from within the document. Whether the directory is eventually + deprecated in favor of in-file definitions is left open (below); this RFC + does not remove anything. + +## Questions or objections + +Recorded as open unless marked otherwise; discussion on the tracking issue and +the draft PR resolves them. + +- **What does `"*"` quantify over? — resolved by the prototype.** The full + valid decision space (constrained Cartesian product), not "all declared + universes." The declared-universes reading is expressible by enumeration, and + a multiverse analysis that silently depended on which configurations happened + to be declared would be fragile. Recorded so the alternative is not silently + re-litigated. +- **Should bare `artifact@*` be allowed in `inputs`? — resolved: no.** Every + fan-out goes through a named, described `Multiverse`. The prototype adopted + this voluntarily and the resulting record is better for it. +- **One collection entry per universe, or per distinct realization? — open.** + When an upstream output ignores some decisions, several member universes share + one realization. The proposal delivers one entry per universe (Steegen-style + counts over universes need the multiplicity) with realizations shared in the + cache; the alternative — collapsing to distinct realizations — is equivalent + to fanning out over the projection of the multiverse onto the output's + `decisions`, and could later be expressed explicitly if needed. +- **Derived universe ids — open.** The proposed scheme (option ids joined by + hyphens in decision-declaration order) is simple and collision-free, but ties + ids to declaration order and can get long for wide decision spaces. Is + declaration-order dependence acceptable, or should the id embed decision ids + (`fertility_assessment=f5,...`) at the cost of length? +- **Sub-analysis scoping — open.** The prototype is a flat analysis. How does + the `@` grammar compose with sub-analyses — can a parent fan out over a + child's decision space, and how do `UniverseNode` selections participate in + `"*"` enumeration? A conservative first cut: `@` references and multiverse + enumeration operate on the current scope's decisions only. +- **Combinatorial guardrails — open.** `"*"` can be astronomically large. + Should validators warn (or runners require confirmation) above a universe + count threshold, or is that purely a tooling concern outside the spec? +- **Intensional multiverses — deferred.** Defining a multiverse by constraint + or sweep (e.g. vary one decision, hold the rest at a named universe) was part + of the original sketch and remains attractive for sensitivity analyses, but + enumerated-plus-`"*"` covers the motivating cases; a constraint language can + be layered on later without breaking this design. +- **Report scoping (RFC-0002 tie-in) — noted.** RFC-0002 left "can a report + compare across universes?" out of scope. Universe-invariant outputs give + reports a natural way to present multiverse-level results without resolving + that question in general. + +## References + +- Steegen, Tuerlinckx, Gelman & Vanpaemel (2016), *Increasing Transparency + Through a Multiverse Analysis*, + [10.1177/1745691616658637](https://doi.org/10.1177/1745691616658637) — the + methodology, and the source of the prototype's reproduced figure. +- Simonsohn, Simmons & Nelson (2020), *Specification curve analysis*, + [10.1038/s41562-020-0912-z](https://doi.org/10.1038/s41562-020-0912-z). +- Liu, Kale, Althoff & Heer (2021), *Boba: Authoring and Visualizing Multiverse + Analyses*, [10.1109/TVCG.2020.3028985](https://doi.org/10.1109/TVCG.2020.3028985) + — a DSL and runtime for decision multiverses. +- [Snakemake `expand()`](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#the-expand-function) + — the workflow-level fan-in pattern over parameter grids. +- [`astra-multiverse-example`](https://github.com/anthonyozerov/astra-multiverse-example) + (Anthony Ozerov) — the working prototype this RFC is grounded in. +- Tracking issue: [#52](https://github.com/LightconeResearch/astra-spec/issues/52). +- [RFC-0002](0002-decouple-reports.md) — establishes element addressing, which + in-file universes and multiverses join as addressable elements. diff --git a/rfcs/README.md b/rfcs/README.md index 173fe457..125b37c2 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -53,4 +53,5 @@ discoverability. Each RFC's `status:` frontmatter field is the durable record. | RFC | Title | Status | |---|---|---| | [0001](0001-establish-the-rfc-process.md) | Establish the ASTRA RFC process and interim governance | Accepted | -| [0002](0002-decouple-reports.md) | Decouple analysis reports from astra.yaml | Active | +| [0002](0002-decouple-reports.md) | Decouple analysis reports from astra.yaml | Accepted | +| [0003](0003-multiverse-analyses.md) | Multiverse analyses — in-file universes and cross-universe artifact references | Draft | From abad787a9baaae48fadd272382b5cb833d839aba Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 30 Jul 2026 11:57:48 -0700 Subject: [PATCH 02/12] RFC-0003: detail universe/multiverse construction grammar; baseline + diffs Universes become diffs against an implicit all-defaults baseline (new 'from:' slot, partial decisions); multiverses gain a base/vary generator form alongside enumeration and '*'; formal reference grammar and field tables added. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 237 +++++++++++++++++++++++-------- 1 file changed, 174 insertions(+), 63 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index ed17bb89..b3d58452 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -34,10 +34,6 @@ The consequence is that a whole class of analyses cannot be declared: - a robustness comparison between two named configurations (`baseline` vs `svm_focused`). -The spec already gestures at the missing machinery: `Output.decisions` -documentation says runners use it to "determine the minimal universe set needed -to materialize the output" — but offers no syntax for an output whose inputs -*span* universes. **Prior art.** Multiverse analysis (Steegen, Tuerlinckx, Gelman & Vanpaemel 2016) and specification-curve analysis (Simonsohn, Simmons & Nelson 2020) are @@ -63,63 +59,134 @@ specification. ## Proposal In plain language: **universes become addressable elements of the analysis -document, sets of universes get names, and an output's inputs may reference an -artifact *at* a universe or *across* a set of universes.** A plain reference -keeps today's meaning — same universe as the consumer — so existing analyses are -untouched. The proposal has three parts. - -### 1. Bring `universes` definitions into `astra.yaml` - -`Analysis` gains an optional **`universes`** slot holding the same `Universe` -objects that today live as standalone files in the `universes/` directory: +document, declared as diffs against a baseline; sets of universes get names and +constructors; and an output's inputs may reference an artifact *at* a universe +or *across* a set of universes.** A plain reference keeps today's meaning — +same universe as the consumer — so existing analyses are untouched. The +proposal has three parts. + +### 1. Bring `universes` into `astra.yaml` — as diffs against a baseline + +`Analysis` gains an optional **`universes`** slot holding `Universe` objects, +and `Universe` gains construction semantics built on two rules: + +- **The baseline.** Every decision already carries a `default` option — whose + doc-string reads "Default option ID for baseline universes." This RFC makes + that normative: the all-defaults selection is the **implicit baseline + universe**, addressable under the reserved id **`baseline`** without being + declared. Declaring a universe with id `baseline` explicitly shadows the + implicit one (to describe it, or to pin a different reference configuration). +- **Universes are diffs.** `Universe.decisions` may be **partial**: unspecified + decisions inherit their selection from the universe's *base*. The base is + named by a new optional **`from`** slot — the same inheritance idiom + `Input`/`Output` already use — and defaults to `baseline`. `from` chains + resolve transitively; cycles are invalid. ```yaml +decisions: + scaling: { default: standard, options: { none: …, standard: …, minmax: … } } + model: { default: random_forest, options: { svm: …, random_forest: …, logistic: … } } + test_size: { default: small, options: { small: …, large: … } } + random_seed: { default: seed_42, options: { seed_42: …, seed_123: … } } + universes: - - id: baseline - description: Default configuration using standard practices - decisions: - scaling: standard - model: random_forest + # `baseline` needs no declaration: it is the all-defaults selection. + - id: svm_focused - description: Configuration optimized for SVM (requires standard scaling) + description: Switch the model to SVM; everything else at baseline. decisions: - scaling: standard - model: svm + model: svm # a one-line diff against `baseline` + + - id: svm_large_test + from: svm_focused # diff against another declared universe + decisions: + test_size: large ``` -The `Universe` class itself is unchanged; what changes is where universes can -live. In-file universes are addressable by the reference grammar below (and by -the RFC-0002 tree-path addressing); the `universes/` directory remains valid for +A universe's **effective selection** is its base's effective selection with the +local `decisions` applied on top. The diff form is not just brevity — it is +record quality: the reader sees exactly what a configuration *deviates from*, +and adding a new decision later does not invalidate every declared universe +(each picks up the new default through its base). A universe that spells out +every decision remains valid, so today's complete `universes/` files carry over +unchanged. + +`Universe` fields after this change: + +| Field | Type | Required | Meaning | +|---|---|---:|---| +| `id` | `string` | Yes | Universe identifier. | +| `description` | `string` | No | Human-readable explanation. | +| `from` | `string` | No | Base universe whose effective selection is inherited. Defaults to `baseline`. | +| `decisions` | map of `decision_id: option_id` | No | Overrides relative to the base; may be complete. | +| `analyses` | map of `UniverseNode` | No | Nested selections mirroring sub-analyses; inheritance applies recursively. | + +In-file universes are addressable by the reference grammar below (and by the +RFC-0002 tree-path addressing); the `universes/` directory remains valid for runner-selected configurations (see *Migration*). -### 2. Add `multiverses` — named sets of universes +### 2. Add `multiverses` — named sets of universes, enumerated or generated `Analysis` gains an optional **`multiverses`** slot. A `Multiverse` names a set of universes so cross-universe steps can reference the set by a stable id and -readers can see, in one place, which decision space a summary quantifies over: +readers can see, in one place, which decision space a summary quantifies over. +Membership takes one of three forms: + +- **(a) Enumeration** — `universes:` lists declared universe ids. +- **(b) Generator** — `vary:` maps each swept decision to `"*"` (all of its + options) or to an explicit option list; the members are the Cartesian product + of the swept option sets, with every *other* decision held at **`base`** (a + universe id, defaulting to `baseline`). Points excluded by `requires` / + `incompatible_with` constraints are dropped. +- **(c) The full space** — `universes: "*"`: every valid point of the decision + space. Sugar for a generator that varies *every* decision over all of its + options (`base` is then irrelevant). ```yaml multiverses: - id: model_robustness description: Configurations relevant to the model-choice robustness check. - universes: [baseline, svm_focused] + universes: [baseline, svm_focused] # (a) enumeration + + - id: scaling_sweep + description: Sensitivity to feature scaling, all else held at baseline. + vary: # (b) generator: 3 universes + scaling: "*" + + - id: seed_scaling_grid + description: Scaling x seed grid around the SVM configuration. + base: svm_focused # (b) generator: 2 x 2 grid + vary: + scaling: [standard, minmax] + random_seed: "*" - id: full_multiverse description: The valid Cartesian product across all decisions. - universes: "*" + universes: "*" # (c) the whole valid space ``` -Membership takes one of two forms: - -- **An enumerated list** of universe ids declared under `universes:`. -- **`"*"`** — every valid point of the decision space: the Cartesian product of - all decisions' options, minus combinations excluded by `requires` / - `incompatible_with` constraints. These grid-point universes are *implicit*: - they need not (and in practice cannot, at 210 of them) be declared - individually. Each receives a **derived id**, formed by joining the selected - option ids in decision-declaration order with hyphens - (e.g. `f5-nmo3-r2-ecl2-ec1`). Option ids are snake_case and may not contain - hyphens, so the derivation is unambiguous and reversible. +Exactly one of `universes` or `vary` must be present; `base` is only meaningful +alongside `vary`. + +**Identity of generated universes.** A generated member is identified by its +*effective selection* — the grid point itself, not the multiverse that produced +it. Its **derived id** joins the selected option ids in decision-declaration +order with hyphens (e.g. `f5-nmo3-r2-ecl2-ec1`; option ids are snake_case and +may not contain hyphens, so the derivation is unambiguous and reversible). The +same point reached through different multiverses — or coinciding with a +declared universe — is *one* universe: realizations and cache keys attach to +the selection, not to the name. Derived ids are valid pin targets +(`f1_score@none-svm-small-seed_42`), though declared names read better. + +`Multiverse` fields: + +| Field | Type | Required | Meaning | +|---|---|---:|---| +| `id` | `string` | Yes | Multiverse identifier. | +| `description` | `string` | No | Human-readable explanation of what the set covers. | +| `universes` | list of universe ids, or `"*"` | One of `universes`/`vary` | Enumerated members, or the full valid decision space. | +| `base` | `string` | No | Generator form: universe holding the non-varied decisions. Defaults to `baseline`. | +| `vary` | map of `decision_id: "*"` \| option list | One of `universes`/`vary` | Generator form: swept decisions and their option sets. | Note the refinement relative to the tracking issue: there, `"*"` was sketched as "all *declared* universes." The prototype clarified that the useful meaning is @@ -132,10 +199,20 @@ of declared universes is expressed by enumerating them. Entries in `Output.inputs` may qualify an artifact reference with a universe or multiverse scope: +``` +input-reference ::= artifact-id [ "@" scope-id ] +scope-id ::= universe-id | multiverse-id +universe-id ::= declared universe id | implicit "baseline" | derived grid-point id +``` + +`artifact-id` keeps its existing grammar (an Input or sibling Output id, +resolving through `from:` chains); `@` cannot appear in element ids today, so +the extension is unambiguous. + | Reference | Meaning | |---|---| | `artifact` | Unchanged: the artifact in the *consumer's own* universe. | -| `artifact@` | The artifact materialized under one declared universe — a **pin**. | +| `artifact@` | The artifact materialized under one specific universe — a **pin**. | | `artifact@` | The artifact under *each* universe in the set — a **fan-out**, resolving to a collection. | Semantics: @@ -226,8 +303,10 @@ multiverses: universes: "*" ``` -**Named-universe comparison** — the iris example, extended with a robustness -check pinned to two declared configurations: +**Baseline, diffs, and a sweep** — the iris example, extended with a pinned +robustness comparison and a one-decision sensitivity sweep. Note that +`baseline` is never declared (it is the all-defaults selection), `svm_focused` +is a one-line diff, and the sweep's domain is a named, described set: ```yaml outputs: @@ -242,18 +321,30 @@ outputs: type: metric description: Baseline vs SVM macro-F1 comparison. inputs: - - f1_score@baseline + - f1_score@baseline # implicit all-defaults universe - f1_score@svm_focused recipe: command: python src/compare.py + - id: scaling_sensitivity + type: metric + description: Macro-F1 as a function of feature scaling, all else at baseline. + inputs: + - f1_score@scaling_sweep # fan-out: one artifact per scaling option + recipe: + command: python src/scaling_sensitivity.py + universes: - - id: baseline - decisions: { scaling: standard, model: random_forest, - test_size: small, random_seed: seed_42 } - id: svm_focused - decisions: { scaling: standard, model: svm, - test_size: small, random_seed: seed_42 } + description: Switch the model to SVM; everything else at baseline. + decisions: + model: svm + +multiverses: + - id: scaling_sweep + description: Sensitivity to feature scaling, all else held at baseline. + vary: + scaling: "*" ``` ## Implementation implications & migration @@ -263,11 +354,14 @@ universes: - `src/astra/schema/analysis.yaml`: add optional `universes` and `multiverses` slots to `Analysis` (the `universe` schema is already imported); extend the `Output.inputs` documentation and validation pattern to admit the `@` - qualifier. -- `src/astra/schema/universe.yaml`: add a `Multiverse` class (`id`, - `description`, `universes: list | "*"`). Revisit the `UniverseNode.universe` - doc-string, which currently hard-codes the "sub-analysis's `universes/` - directory." + qualifier; update the `Decision.default` doc-string to its normative role as + the baseline selection. +- `src/astra/schema/universe.yaml`: add the `from` slot to `Universe` and + document `decisions` as overrides relative to the base; add a `Multiverse` + class (`id`, `description`, `universes`, `base`, `vary`) with a rule making + `universes` and `vary` mutually exclusive. Revisit the + `UniverseNode.universe` doc-string, which currently hard-codes the + "sub-analysis's `universes/` directory." - Generated artifacts: `just gen-python`, `just gen-doc`; the published JSON Schema at `astra-spec.org//schema/…` shifts, which is the propagation point for both SDKs. @@ -279,10 +373,13 @@ universes: **`astra-tools` (Python CLI + SDK):** -- `astra validate` gains checks: every `@` target resolves to a declared - universe or multiverse id; declared universes select valid options and honor - `requires` / `incompatible_with`; enumerated multiverse members exist; a - pinned reference targets an output that is active in the pinned universe. +- `astra validate` gains checks: every `@` target resolves to a universe + (declared, `baseline`, or derived) or multiverse id; `from` chains resolve + and are acyclic; a universe relying on baseline completion errors if some + decision lacks a `default`; effective selections select valid options and + honor `requires` / `incompatible_with`; `vary` names existing decisions and + options; enumerated multiverse members exist; a pinned reference targets an + output that is active in the pinned universe. - Runner semantics: cross-universe edges change scheduling — one output can now demand materialization of an upstream artifact under many universes — and the derived-id scheme for `"*"` grid points must match the spec so artifact paths @@ -296,7 +393,9 @@ SDK parses `Output.inputs`. - Plain references keep their exact current meaning, and both new slots are optional, so existing analyses validate unchanged: the schema change is - **additive — a minor bump** under the versioning policy. + **additive — a minor bump** under the versioning policy. Allowing partial + `Universe.decisions` is a loosening: complete universe files stay valid as-is + (an empty diff), and partial ones only become *newly* valid. - The `universes/` directory remains supported as a place for runner-selected configurations; in-file `universes` are required only when a universe must be *referenced* from within the document. Whether the directory is eventually @@ -329,6 +428,18 @@ the draft PR resolves them. ids to declaration order and can get long for wide decision spaces. Is declaration-order dependence acceptable, or should the id embed decision ids (`fertility_assessment=f5,...`) at the cost of length? +- **Is reserving `baseline` acceptable? — open.** The implicit baseline makes + the common case free but claims an id and silently changes meaning if a + project declares its own `baseline` with non-default selections. Alternatives: + require an explicit declaration to enable `@baseline`, or a distinct spelling + (`@~` or `@default`) for the all-defaults universe. The proposal leans on the + reserved id because `Decision.default`'s own doc-string already promises it. +- **Are diffs too implicit? — open.** With `from:` chains, a universe's full + selection is no longer visible at its declaration site. The record arguably + *improves* (deviation-from-baseline is the scientifically meaningful datum, + and new decisions propagate through defaults instead of invalidating every + universe), but tooling should make the effective selection one command away + (e.g. `astra universe show `). - **Sub-analysis scoping — open.** The prototype is a flat analysis. How does the `@` grammar compose with sub-analyses — can a parent fan out over a child's decision space, and how do `UniverseNode` selections participate in @@ -337,11 +448,11 @@ the draft PR resolves them. - **Combinatorial guardrails — open.** `"*"` can be astronomically large. Should validators warn (or runners require confirmation) above a universe count threshold, or is that purely a tooling concern outside the spec? -- **Intensional multiverses — deferred.** Defining a multiverse by constraint - or sweep (e.g. vary one decision, hold the rest at a named universe) was part - of the original sketch and remains attractive for sensitivity analyses, but - enumerated-plus-`"*"` covers the motivating cases; a constraint language can - be layered on later without breaking this design. +- **Intensional multiverses — partially resolved.** The `base`/`vary` generator + covers the sweep and grid cases from the original sketch (vary some decisions, + hold the rest at a named universe). A general *constraint* language — "all + valid universes where `scaling` is not `none`" — is still deferred; it can be + layered on later (e.g. a `where:` clause) without breaking this design. - **Report scoping (RFC-0002 tie-in) — noted.** RFC-0002 left "can a report compare across universes?" out of scope. Universe-invariant outputs give reports a natural way to present multiverse-level results without resolving From c68b296fb19a709fbfa33cc812da0c75cb803781 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 30 Jul 2026 12:08:33 -0700 Subject: [PATCH 03/12] RFC-0003: remove privileged baseline; no universe is special Universes are declared in full or as explicit diffs of one another (from: required for partial declarations); generators require an explicit base when vary is partial; Decision.default is demoted to a presentational hint. Records the rejected implicit-baseline design and grounds the principle in the multiverse/PCS stance (Yu & Kumbier 2020). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 209 ++++++++++++++++--------------- 1 file changed, 105 insertions(+), 104 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index b3d58452..6d44cdad 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -31,84 +31,67 @@ The consequence is that a whole class of analyses cannot be declared: analysis, Steegen et al. 2016), - a specification-curve figure over the full decision grid (Simonsohn et al. 2020), -- a robustness comparison between two named configurations (`baseline` vs - `svm_focused`). - - -**Prior art.** Multiverse analysis (Steegen, Tuerlinckx, Gelman & Vanpaemel -2016) and specification-curve analysis (Simonsohn, Simmons & Nelson 2020) are -established methodology; tooling such as [Boba](https://github.com/uwdata/boba) -(Liu et al. 2021) demonstrates a DSL for authoring and executing decision -multiverses. On the workflow side, Snakemake's `expand()` is the standard -pattern for fanning a rule's inputs over a parameter grid — the same fan-in -shape this RFC needs at the specification level. - -**Working prototype.** The mechanism proposed here is demonstrated end-to-end by -[`astra-multiverse-example`](https://github.com/anthonyozerov/astra-multiverse-example) -(Anthony Ozerov), an independent reproduction of Figure 1 of Steegen et -al. 2016. Its `astra.yaml` declares the paper's five data-processing decisions; -option constraints (`incompatible_with`) cut the Cartesian product down to 210 -valid universes, and a `when` condition on the Study 1 output reduces *its* -active set to 120. A single `multiverses` entry names the full valid decision -space, six metric outputs are referenced as `@full_multiverse` by a -figure output, and a runner executes all universes and reproduces the paper's -published significance counts. As with RFC-0002, this RFC is grounded in -something that works; its job is to decide what of it belongs in the ASTRA -specification. +- a stability comparison between two named configurations (a random-forest vs + an SVM specification of the same pipeline). ## Proposal In plain language: **universes become addressable elements of the analysis -document, declared as diffs against a baseline; sets of universes get names and -constructors; and an output's inputs may reference an artifact *at* a universe -or *across* a set of universes.** A plain reference keeps today's meaning — -same universe as the consumer — so existing analyses are untouched. The -proposal has three parts. - -### 1. Bring `universes` into `astra.yaml` — as diffs against a baseline +document, declared in full or as diffs of one another; sets of universes get +names and constructors; and an output's inputs may reference an artifact *at* +a universe or *across* a set of universes.** A plain reference keeps today's +meaning — same universe as the consumer — so existing analyses are untouched. +The proposal has three parts. + +One design principle runs through all three: **no universe is privileged by the +spec.** The premise of multiverse analysis — and of the stability principle in +the PCS framework (Yu & Kumbier 2020) — is that every defensible specification +has equal standing; a spec-blessed "default" or "baseline" universe would +invite reading one path as *the* analysis and the rest as robustness garnish. +ASTRA therefore defines no implicit universe and reserves no universe id: +every universe and every relationship between universes is declared by the +author. (`Decision.default` remains a presentational hint for scaffolding and +editors; it plays no role in universe construction.) + +### 1. Bring `universes` into `astra.yaml` — in full or as diffs of one another `Analysis` gains an optional **`universes`** slot holding `Universe` objects, -and `Universe` gains construction semantics built on two rules: - -- **The baseline.** Every decision already carries a `default` option — whose - doc-string reads "Default option ID for baseline universes." This RFC makes - that normative: the all-defaults selection is the **implicit baseline - universe**, addressable under the reserved id **`baseline`** without being - declared. Declaring a universe with id `baseline` explicitly shadows the - implicit one (to describe it, or to pin a different reference configuration). -- **Universes are diffs.** `Universe.decisions` may be **partial**: unspecified - decisions inherit their selection from the universe's *base*. The base is - named by a new optional **`from`** slot — the same inheritance idiom - `Input`/`Output` already use — and defaults to `baseline`. `from` chains - resolve transitively; cycles are invalid. +and `Universe` gains one construction rule: -```yaml -decisions: - scaling: { default: standard, options: { none: …, standard: …, minmax: … } } - model: { default: random_forest, options: { svm: …, random_forest: …, logistic: … } } - test_size: { default: small, options: { small: …, large: … } } - random_seed: { default: seed_42, options: { seed_42: …, seed_123: … } } +- **Universes may be declared as diffs.** `Universe.decisions` may be + **partial** when a new optional **`from`** slot — the same inheritance idiom + `Input`/`Output` already use — names another declared universe: unspecified + decisions inherit their selection from that base. Without `from`, `decisions` + must be complete, exactly as today. `from` chains resolve transitively; + cycles are invalid. `from` expresses a *relationship between two declared + universes*, chosen by the author — it does not anoint the base as special. +```yaml universes: - # `baseline` needs no declaration: it is the all-defaults selection. + - id: rf_standard + description: Random forest with standard scaling. + decisions: + scaling: standard + model: random_forest + test_size: small + random_seed: seed_42 - - id: svm_focused - description: Switch the model to SVM; everything else at baseline. + - id: svm_standard + from: rf_standard # same specification, one decision changed decisions: - model: svm # a one-line diff against `baseline` + model: svm - id: svm_large_test - from: svm_focused # diff against another declared universe + from: svm_standard # diffs chain decisions: test_size: large ``` A universe's **effective selection** is its base's effective selection with the local `decisions` applied on top. The diff form is not just brevity — it is -record quality: the reader sees exactly what a configuration *deviates from*, -and adding a new decision later does not invalidate every declared universe -(each picks up the new default through its base). A universe that spells out -every decision remains valid, so today's complete `universes/` files carry over +record quality: it states *how two specifications relate*, which is exactly the +datum a stability comparison rests on. A universe that spells out every +decision remains valid, so today's complete `universes/` files carry over unchanged. `Universe` fields after this change: @@ -117,8 +100,8 @@ unchanged. |---|---|---:|---| | `id` | `string` | Yes | Universe identifier. | | `description` | `string` | No | Human-readable explanation. | -| `from` | `string` | No | Base universe whose effective selection is inherited. Defaults to `baseline`. | -| `decisions` | map of `decision_id: option_id` | No | Overrides relative to the base; may be complete. | +| `from` | `string` | No | Declared universe whose effective selection is inherited. | +| `decisions` | map of `decision_id: option_id` | No | Overrides relative to `from`; must be complete when `from` is absent. | | `analyses` | map of `UniverseNode` | No | Nested selections mirroring sub-analyses; inheritance applies recursively. | In-file universes are addressable by the reference grammar below (and by the @@ -135,27 +118,29 @@ Membership takes one of three forms: - **(a) Enumeration** — `universes:` lists declared universe ids. - **(b) Generator** — `vary:` maps each swept decision to `"*"` (all of its options) or to an explicit option list; the members are the Cartesian product - of the swept option sets, with every *other* decision held at **`base`** (a - universe id, defaulting to `baseline`). Points excluded by `requires` / - `incompatible_with` constraints are dropped. + of the swept option sets. Any decision *not* listed in `vary` is held at + **`base`**, a declared universe id — required whenever `vary` does not cover + every decision, so nothing is pinned implicitly. Points excluded by + `requires` / `incompatible_with` constraints are dropped. - **(c) The full space** — `universes: "*"`: every valid point of the decision space. Sugar for a generator that varies *every* decision over all of its - options (`base` is then irrelevant). + options (no `base` — nothing is held fixed). ```yaml multiverses: - - id: model_robustness - description: Configurations relevant to the model-choice robustness check. - universes: [baseline, svm_focused] # (a) enumeration + - id: model_stability + description: Configurations for the model-choice stability comparison. + universes: [rf_standard, svm_standard] # (a) enumeration - id: scaling_sweep - description: Sensitivity to feature scaling, all else held at baseline. - vary: # (b) generator: 3 universes + description: Vary feature scaling; all else held at rf_standard. + base: rf_standard # (b) generator: 3 universes + vary: scaling: "*" - id: seed_scaling_grid description: Scaling x seed grid around the SVM configuration. - base: svm_focused # (b) generator: 2 x 2 grid + base: svm_standard # (b) generator: 2 x 2 grid vary: scaling: [standard, minmax] random_seed: "*" @@ -165,8 +150,8 @@ multiverses: universes: "*" # (c) the whole valid space ``` -Exactly one of `universes` or `vary` must be present; `base` is only meaningful -alongside `vary`. +Exactly one of `universes` or `vary` must be present; `base` accompanies `vary` +and is required unless `vary` covers every decision in scope. **Identity of generated universes.** A generated member is identified by its *effective selection* — the grid point itself, not the multiverse that produced @@ -185,7 +170,7 @@ the selection, not to the name. Derived ids are valid pin targets | `id` | `string` | Yes | Multiverse identifier. | | `description` | `string` | No | Human-readable explanation of what the set covers. | | `universes` | list of universe ids, or `"*"` | One of `universes`/`vary` | Enumerated members, or the full valid decision space. | -| `base` | `string` | No | Generator form: universe holding the non-varied decisions. Defaults to `baseline`. | +| `base` | `string` | With partial `vary` | Generator form: declared universe holding the non-varied decisions. | | `vary` | map of `decision_id: "*"` \| option list | One of `universes`/`vary` | Generator form: swept decisions and their option sets. | Note the refinement relative to the tracking issue: there, `"*"` was sketched as @@ -202,7 +187,7 @@ multiverse scope: ``` input-reference ::= artifact-id [ "@" scope-id ] scope-id ::= universe-id | multiverse-id -universe-id ::= declared universe id | implicit "baseline" | derived grid-point id +universe-id ::= declared universe id | derived grid-point id ``` `artifact-id` keeps its existing grammar (an Input or sibling Output id, @@ -303,10 +288,10 @@ multiverses: universes: "*" ``` -**Baseline, diffs, and a sweep** — the iris example, extended with a pinned -robustness comparison and a one-decision sensitivity sweep. Note that -`baseline` is never declared (it is the all-defaults selection), `svm_focused` -is a one-line diff, and the sweep's domain is a named, described set: +**Declared universes, diffs, and a sweep** — the iris example, extended with a +pinned stability comparison and a one-decision sweep. `svm_standard` is a +one-line diff of `rf_standard` (a relationship, not a hierarchy — neither is +privileged), and every set an output quantifies over is named and described: ```yaml outputs: @@ -319,30 +304,37 @@ outputs: - id: model_comparison type: metric - description: Baseline vs SVM macro-F1 comparison. + description: Macro-F1 under the random-forest vs the SVM specification. inputs: - - f1_score@baseline # implicit all-defaults universe - - f1_score@svm_focused + - f1_score@rf_standard + - f1_score@svm_standard recipe: command: python src/compare.py - id: scaling_sensitivity type: metric - description: Macro-F1 as a function of feature scaling, all else at baseline. + description: Macro-F1 as a function of feature scaling, all else at rf_standard. inputs: - f1_score@scaling_sweep # fan-out: one artifact per scaling option recipe: command: python src/scaling_sensitivity.py universes: - - id: svm_focused - description: Switch the model to SVM; everything else at baseline. + - id: rf_standard + description: Random forest with standard scaling. + decisions: { scaling: standard, model: random_forest, + test_size: small, random_seed: seed_42 } + + - id: svm_standard + description: Same specification with the model switched to SVM. + from: rf_standard decisions: model: svm multiverses: - id: scaling_sweep - description: Sensitivity to feature scaling, all else held at baseline. + description: Vary feature scaling; all else held at rf_standard. + base: rf_standard vary: scaling: "*" ``` @@ -354,8 +346,9 @@ multiverses: - `src/astra/schema/analysis.yaml`: add optional `universes` and `multiverses` slots to `Analysis` (the `universe` schema is already imported); extend the `Output.inputs` documentation and validation pattern to admit the `@` - qualifier; update the `Decision.default` doc-string to its normative role as - the baseline selection. + qualifier; clarify in the `Decision.default` doc-string that the default is a + presentational/scaffolding hint with no role in universe construction (its + current "for baseline universes" wording suggests otherwise). - `src/astra/schema/universe.yaml`: add the `from` slot to `Universe` and document `decisions` as overrides relative to the base; add a `Multiverse` class (`id`, `description`, `universes`, `base`, `vary`) with a rule making @@ -374,12 +367,12 @@ multiverses: **`astra-tools` (Python CLI + SDK):** - `astra validate` gains checks: every `@` target resolves to a universe - (declared, `baseline`, or derived) or multiverse id; `from` chains resolve - and are acyclic; a universe relying on baseline completion errors if some - decision lacks a `default`; effective selections select valid options and - honor `requires` / `incompatible_with`; `vary` names existing decisions and - options; enumerated multiverse members exist; a pinned reference targets an - output that is active in the pinned universe. + (declared or derived) or multiverse id; `from` chains resolve and are + acyclic; a universe without `from` selects an option for every decision; + effective selections select valid options and honor `requires` / + `incompatible_with`; `vary` names existing decisions and options, and `base` + is present whenever `vary` is partial; enumerated multiverse members exist; + a pinned reference targets an output that is active in the pinned universe. - Runner semantics: cross-universe edges change scheduling — one output can now demand materialization of an upstream artifact under many universes — and the derived-id scheme for `"*"` grid points must match the spec so artifact paths @@ -428,18 +421,22 @@ the draft PR resolves them. ids to declaration order and can get long for wide decision spaces. Is declaration-order dependence acceptable, or should the id embed decision ids (`fertility_assessment=f5,...`) at the cost of length? -- **Is reserving `baseline` acceptable? — open.** The implicit baseline makes - the common case free but claims an id and silently changes meaning if a - project declares its own `baseline` with non-default selections. Alternatives: - require an explicit declaration to enable `@baseline`, or a distinct spelling - (`@~` or `@default`) for the all-defaults universe. The proposal leans on the - reserved id because `Decision.default`'s own doc-string already promises it. +- **Should decision defaults define an implicit `baseline` universe? — + resolved: no.** An earlier draft derived an implicit, spec-reserved + `baseline` universe from `Decision.default` and made it the default base for + diffs and generators. Rejected: privileging one specification is contrary to + the premise of multiverse analysis and to the stability principle of the PCS + framework (Yu & Kumbier 2020) — every defensible universe has equal standing, + and a spec-blessed default invites treating one path as *the* analysis with + the rest as robustness garnish. Consequently: no reserved universe ids; + partial universes require an explicit `from`; generators require an explicit + `base` when `vary` is partial; `Decision.default` stays a presentational + hint. Recorded so the convenience argument is not silently re-litigated. - **Are diffs too implicit? — open.** With `from:` chains, a universe's full selection is no longer visible at its declaration site. The record arguably - *improves* (deviation-from-baseline is the scientifically meaningful datum, - and new decisions propagate through defaults instead of invalidating every - universe), but tooling should make the effective selection one command away - (e.g. `astra universe show `). + *improves* — the relationship between specifications is the datum a stability + comparison rests on — but tooling should make the effective selection one + command away (e.g. `astra universe show `). - **Sub-analysis scoping — open.** The prototype is a flat analysis. How does the `@` grammar compose with sub-analyses — can a parent fan out over a child's decision space, and how do `UniverseNode` selections participate in @@ -469,6 +466,10 @@ the draft PR resolves them. - Liu, Kale, Althoff & Heer (2021), *Boba: Authoring and Visualizing Multiverse Analyses*, [10.1109/TVCG.2020.3028985](https://doi.org/10.1109/TVCG.2020.3028985) — a DSL and runtime for decision multiverses. +- Yu & Kumbier (2020), *Veridical data science*, + [10.1073/pnas.1901326117](https://doi.org/10.1073/pnas.1901326117) — the PCS + framework; its stability principle motivates treating all defensible + specifications symmetrically, which is why this RFC privileges no universe. - [Snakemake `expand()`](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#the-expand-function) — the workflow-level fan-in pattern over parameter grids. - [`astra-multiverse-example`](https://github.com/anthonyozerov/astra-multiverse-example) From 9c69c7bf0e95a78bb67c9a22744334c8b1e64921 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 30 Jul 2026 12:12:45 -0700 Subject: [PATCH 04/12] RFC-0003: tighten edge-case semantics from review pass - @ qualifier restricted to Output references (external Inputs don't vary) - universe scoping defined recursively: decisions or unqualified refs to scoped artifacts propagate it; @ cuts it; invariant recipes forbid {decisions.*} - multiverses are sets; when-deactivated decisions make the space a tree (inactive decisions omitted from selections and derived ids) - shared universe/multiverse id namespace; declared ids may not spell a different selection's derived id - only in-file universes are addressable; enumeration accepts derived ids - validator checklist synced with all of the above Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 68 +++++++++++++++++++++++--------- 1 file changed, 49 insertions(+), 19 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index 6d44cdad..9aca5c33 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -104,9 +104,11 @@ unchanged. | `decisions` | map of `decision_id: option_id` | No | Overrides relative to `from`; must be complete when `from` is absent. | | `analyses` | map of `UniverseNode` | No | Nested selections mirroring sub-analyses; inheritance applies recursively. | -In-file universes are addressable by the reference grammar below (and by the -RFC-0002 tree-path addressing); the `universes/` directory remains valid for -runner-selected configurations (see *Migration*). +Only in-file universes are addressable: `from`, `base`, multiverse +enumerations, and `@` scopes resolve against the document's `universes` (plus +derived grid-point ids, below) — never against `universes/` directory files, +which remain valid for runner-selected configurations (see *Migration*). In-file +universes also join the RFC-0002 tree-path addressing. ### 2. Add `multiverses` — named sets of universes, enumerated or generated @@ -115,7 +117,8 @@ of universes so cross-universe steps can reference the set by a stable id and readers can see, in one place, which decision space a summary quantifies over. Membership takes one of three forms: -- **(a) Enumeration** — `universes:` lists declared universe ids. +- **(a) Enumeration** — `universes:` lists universe ids (declared, or derived + grid-point ids). - **(b) Generator** — `vary:` maps each swept decision to `"*"` (all of its options) or to an explicit option list; the members are the Cartesian product of the swept option sets. Any decision *not* listed in `vary` is held at @@ -163,13 +166,30 @@ declared universe — is *one* universe: realizations and cache keys attach to the selection, not to the name. Derived ids are valid pin targets (`f1_score@none-svm-small-seed_42`), though declared names read better. +Three resolution rules keep this well-defined: + +- **Multiverses are sets.** Members with identical effective selections + collapse to one — whether listed twice in an enumeration or reached twice by + a generator. +- **The decision space is a tree, not a grid.** A `when` condition can + deactivate a decision under some selections; enumeration recurses over + *active* decisions only, and an inactive decision is omitted from the + effective selection and the derived id. Points that differ only in inactive + decisions are the same universe. +- **One namespace, no ambiguity.** Universe and multiverse ids share a + namespace, so a `scope-id` resolves without guessing; a collision between + them is invalid. A declared universe id that spells the derived id of a + *different* selection is likewise invalid (checkable: parse the id as an + option join and compare) — a declared id may only coincide with the derived + id of its own selection, where the two readings agree. + `Multiverse` fields: | Field | Type | Required | Meaning | |---|---|---:|---| | `id` | `string` | Yes | Multiverse identifier. | | `description` | `string` | No | Human-readable explanation of what the set covers. | -| `universes` | list of universe ids, or `"*"` | One of `universes`/`vary` | Enumerated members, or the full valid decision space. | +| `universes` | list of universe ids, or `"*"` | One of `universes`/`vary` | Enumerated members (declared or derived ids), or the full valid decision space. | | `base` | `string` | With partial `vary` | Generator form: declared universe holding the non-varied decisions. | | `vary` | map of `decision_id: "*"` \| option list | One of `universes`/`vary` | Generator form: swept decisions and their option sets. | @@ -192,7 +212,9 @@ universe-id ::= declared universe id | derived grid-point id `artifact-id` keeps its existing grammar (an Input or sibling Output id, resolving through `from:` chains); `@` cannot appear in element ids today, so -the extension is unambiguous. +the extension is unambiguous. The qualifier is only valid on references that +resolve to an *Output*: an external `Input` does not vary by universe, so +qualifying it is an error. | Reference | Meaning | |---|---| @@ -207,12 +229,16 @@ Semantics: *active* — universes excluded by the output's `when` conditions contribute nothing. (In the prototype, `religiosity_study1_p@full_multiverse` yields 120 artifacts while its siblings yield 210.) -- **Universe-invariant consumers.** An output whose inputs are all pinned or - fanned-out does not itself vary with the current universe: it is materialized - once per project, not once per universe — the right identity for a - multiverse-level summary or figure. An output that mixes plain and qualified - references remains universe-scoped and additionally pulls in the referenced - cross-universe artifacts. +- **Universe scoping is inferred, and `@` cuts it.** An output is + *universe-scoped* if it declares `decisions`, or if any unqualified input + reference resolves to a universe-scoped artifact. Qualified references never + propagate scope — the qualifier fixes the universe(s) — and external + `Input`s carry none. An output that comes out universe-invariant is + materialized once per project, not once per universe: the right identity for + a multiverse-level summary or figure. Its recipe consequently has no + "current universe," so `{decisions.}` placeholders are invalid there; + and since qualifying a reference to a universe-invariant artifact selects + nothing, validators should flag it as redundant. - **Recipe surface.** In recipe templates, `{inputs.}` for a fan-out reference expands to the collection of materialized artifact paths; how the collection is surfaced (space-separated paths, a manifest file, a sidecar) @@ -366,13 +392,17 @@ multiverses: **`astra-tools` (Python CLI + SDK):** -- `astra validate` gains checks: every `@` target resolves to a universe - (declared or derived) or multiverse id; `from` chains resolve and are - acyclic; a universe without `from` selects an option for every decision; - effective selections select valid options and honor `requires` / - `incompatible_with`; `vary` names existing decisions and options, and `base` - is present whenever `vary` is partial; enumerated multiverse members exist; - a pinned reference targets an output that is active in the pinned universe. +- `astra validate` gains checks: universe and multiverse ids are unique in + their shared namespace, and no declared id spells the derived id of a + different selection; every `@` target resolves (declared or derived + universe, or multiverse) and qualifies a reference to an Output; `from` + chains resolve and are acyclic; a universe without `from` selects an option + for every *active* decision; effective selections select valid options and + honor `requires` / `incompatible_with`; `vary` names existing decisions and + options, `base` is present whenever `vary` is partial, and a generator with + no valid members is an error; enumerated multiverse members exist; a pinned + reference targets an output that is active in the pinned universe; a + universe-invariant output's recipe uses no `{decisions.}` placeholders. - Runner semantics: cross-universe edges change scheduling — one output can now demand materialization of an upstream artifact under many universes — and the derived-id scheme for `"*"` grid points must match the spec so artifact paths From 7bf4f0783662f195f899de635e66bdcc7be646f9 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Thu, 30 Jul 2026 12:13:09 -0700 Subject: [PATCH 05/12] RFC-0003: deduplicate the '*' rationale (kept in Questions) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index 9aca5c33..086c65a9 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -193,11 +193,8 @@ Three resolution rules keep this well-defined: | `base` | `string` | With partial `vary` | Generator form: declared universe holding the non-varied decisions. | | `vary` | map of `decision_id: "*"` \| option list | One of `universes`/`vary` | Generator form: swept decisions and their option sets. | -Note the refinement relative to the tracking issue: there, `"*"` was sketched as -"all *declared* universes." The prototype clarified that the useful meaning is -the **full valid decision space** — multiverse analyses quantify over the -choices themselves, not over whichever configurations happen to be named. A set -of declared universes is expressed by enumerating them. +(`"*"` deliberately means the full valid decision space, not "all declared +universes" as sketched in the tracking issue — see *Questions* for the record.) ### 3. Extend the artifact reference grammar with `@` From e544042bd0c16ff6279b5fb4bcf36b2312866eb3 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Wed, 5 Aug 2026 14:51:25 +0200 Subject: [PATCH 06/12] RFC-0003: unify universe/generator diffing on 'base' 'from' and 'base' were one concept with two names; worse, ASTRA's existing 'from' means pure re-export aliasing (no overrides), while universe diffing is inherit-and-override. Rename Universe.from to base, note the deliberate contrast with the alias idiom, and state the generator equivalence: vary is decisions generalized to option sets. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 47 ++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 18 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index 086c65a9..d6799a07 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -4,6 +4,8 @@ title: Multiverse analyses — in-file universes and cross-universe artifact ref status: Draft # Draft | Active | Accepted | Rejected | Superseded authors: - Francois Lanusse (@eiffl) + - Anthony Ozerov (@anthonyozerov) + - Jacopo Teneggi (@JacopoTeneggi) created: 2026-07-30 tracking-issue: https://github.com/LightconeResearch/astra-spec/issues/52 superseded-by: @@ -59,12 +61,15 @@ editors; it plays no role in universe construction.) and `Universe` gains one construction rule: - **Universes may be declared as diffs.** `Universe.decisions` may be - **partial** when a new optional **`from`** slot — the same inheritance idiom - `Input`/`Output` already use — names another declared universe: unspecified - decisions inherit their selection from that base. Without `from`, `decisions` - must be complete, exactly as today. `from` chains resolve transitively; - cycles are invalid. `from` expresses a *relationship between two declared - universes*, chosen by the author — it does not anoint the base as special. + **partial** when a new optional **`base`** slot names another declared + universe: unspecified decisions inherit their selection from the base. + Without `base`, `decisions` must be complete, exactly as today. `base` chains + resolve transitively; cycles are invalid. `base` expresses a *relationship + between two declared universes*, chosen by the author — it does not anoint + the base as special. (The existing `from` idiom is deliberately *not* reused: + on `Input`/`Output`, `from` is a pure re-export alias that forbids local + overrides, whereas `base` is inherit-*and*-override — a different operation + deserving a different word.) ```yaml universes: @@ -77,12 +82,12 @@ universes: random_seed: seed_42 - id: svm_standard - from: rf_standard # same specification, one decision changed + base: rf_standard # same specification, one decision changed decisions: model: svm - id: svm_large_test - from: svm_standard # diffs chain + base: svm_standard # diffs chain decisions: test_size: large ``` @@ -100,12 +105,12 @@ unchanged. |---|---|---:|---| | `id` | `string` | Yes | Universe identifier. | | `description` | `string` | No | Human-readable explanation. | -| `from` | `string` | No | Declared universe whose effective selection is inherited. | -| `decisions` | map of `decision_id: option_id` | No | Overrides relative to `from`; must be complete when `from` is absent. | +| `base` | `string` | No | Declared universe whose effective selection is inherited. | +| `decisions` | map of `decision_id: option_id` | No | Overrides relative to `base`; must be complete when `base` is absent. | | `analyses` | map of `UniverseNode` | No | Nested selections mirroring sub-analyses; inheritance applies recursively. | -Only in-file universes are addressable: `from`, `base`, multiverse -enumerations, and `@` scopes resolve against the document's `universes` (plus +Only in-file universes are addressable: `base` (on universes and on multiverse +generators), enumerations, and `@` scopes resolve against the document's `universes` (plus derived grid-point ids, below) — never against `universes/` directory files, which remain valid for runner-selected configurations (see *Migration*). In-file universes also join the RFC-0002 tree-path addressing. @@ -156,6 +161,12 @@ multiverses: Exactly one of `universes` or `vary` must be present; `base` accompanies `vary` and is required unless `vary` covers every decision in scope. +`base` here is the *same* diff mechanism a `Universe` uses, and `vary` is +`decisions` generalized from single options to option sets: a generator is +exactly the set of diff-universes `{base: , decisions: }` for +every valid point of the swept product. One construction, scalar or +set-valued. + **Identity of generated universes.** A generated member is identified by its *effective selection* — the grid point itself, not the multiverse that produced it. Its **derived id** joins the selected option ids in decision-declaration @@ -350,7 +361,7 @@ universes: - id: svm_standard description: Same specification with the model switched to SVM. - from: rf_standard + base: rf_standard decisions: model: svm @@ -372,7 +383,7 @@ multiverses: qualifier; clarify in the `Decision.default` doc-string that the default is a presentational/scaffolding hint with no role in universe construction (its current "for baseline universes" wording suggests otherwise). -- `src/astra/schema/universe.yaml`: add the `from` slot to `Universe` and +- `src/astra/schema/universe.yaml`: add the `base` slot to `Universe` and document `decisions` as overrides relative to the base; add a `Multiverse` class (`id`, `description`, `universes`, `base`, `vary`) with a rule making `universes` and `vary` mutually exclusive. Revisit the @@ -392,8 +403,8 @@ multiverses: - `astra validate` gains checks: universe and multiverse ids are unique in their shared namespace, and no declared id spells the derived id of a different selection; every `@` target resolves (declared or derived - universe, or multiverse) and qualifies a reference to an Output; `from` - chains resolve and are acyclic; a universe without `from` selects an option + universe, or multiverse) and qualifies a reference to an Output; `base` + chains resolve and are acyclic; a universe without `base` selects an option for every *active* decision; effective selections select valid options and honor `requires` / `incompatible_with`; `vary` names existing decisions and options, `base` is present whenever `vary` is partial, and a generator with @@ -456,10 +467,10 @@ the draft PR resolves them. framework (Yu & Kumbier 2020) — every defensible universe has equal standing, and a spec-blessed default invites treating one path as *the* analysis with the rest as robustness garnish. Consequently: no reserved universe ids; - partial universes require an explicit `from`; generators require an explicit + partial universes require an explicit `base`; generators require an explicit `base` when `vary` is partial; `Decision.default` stays a presentational hint. Recorded so the convenience argument is not silently re-litigated. -- **Are diffs too implicit? — open.** With `from:` chains, a universe's full +- **Are diffs too implicit? — open.** With `base` chains, a universe's full selection is no longer visible at its declaration site. The record arguably *improves* — the relationship between specifications is the datum a stability comparison rests on — but tooling should make the effective selection one From b50505e16945e384112ff4e736a9192f2cfd49ed Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Wed, 5 Aug 2026 14:56:43 +0200 Subject: [PATCH 07/12] RFC-0003: generated universes are anonymous; drop addressable derived ids The option-join id grows unboundedly for '*' over wide decision spaces and leaks into every artifact path; making it addressable also forced collision/shadowing rules. Identity now rests solely on the effective selection; addressing a grid point means declaring it (a three-line base diff); storage spelling (join vs digest) becomes a runner convention. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 83 ++++++++++++++++---------------- 1 file changed, 41 insertions(+), 42 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index d6799a07..d0b1baca 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -110,10 +110,10 @@ unchanged. | `analyses` | map of `UniverseNode` | No | Nested selections mirroring sub-analyses; inheritance applies recursively. | Only in-file universes are addressable: `base` (on universes and on multiverse -generators), enumerations, and `@` scopes resolve against the document's `universes` (plus -derived grid-point ids, below) — never against `universes/` directory files, -which remain valid for runner-selected configurations (see *Migration*). In-file -universes also join the RFC-0002 tree-path addressing. +generators), enumerations, and `@` scopes resolve against the document's +`universes` — never against `universes/` directory files, which remain valid +for runner-selected configurations (see *Migration*). In-file universes also +join the RFC-0002 tree-path addressing. ### 2. Add `multiverses` — named sets of universes, enumerated or generated @@ -122,8 +122,7 @@ of universes so cross-universe steps can reference the set by a stable id and readers can see, in one place, which decision space a summary quantifies over. Membership takes one of three forms: -- **(a) Enumeration** — `universes:` lists universe ids (declared, or derived - grid-point ids). +- **(a) Enumeration** — `universes:` lists declared universe ids. - **(b) Generator** — `vary:` maps each swept decision to `"*"` (all of its options) or to an explicit option list; the members are the Cartesian product of the swept option sets. Any decision *not* listed in `vary` is held at @@ -161,21 +160,19 @@ multiverses: Exactly one of `universes` or `vary` must be present; `base` accompanies `vary` and is required unless `vary` covers every decision in scope. -`base` here is the *same* diff mechanism a `Universe` uses, and `vary` is -`decisions` generalized from single options to option sets: a generator is -exactly the set of diff-universes `{base: , decisions: }` for -every valid point of the swept product. One construction, scalar or -set-valued. - **Identity of generated universes.** A generated member is identified by its *effective selection* — the grid point itself, not the multiverse that produced -it. Its **derived id** joins the selected option ids in decision-declaration -order with hyphens (e.g. `f5-nmo3-r2-ecl2-ec1`; option ids are snake_case and -may not contain hyphens, so the derivation is unambiguous and reversible). The -same point reached through different multiverses — or coinciding with a -declared universe — is *one* universe: realizations and cache keys attach to -the selection, not to the name. Derived ids are valid pin targets -(`f1_score@none-svm-small-seed_42`), though declared names read better. +it. The same point reached through different multiverses — or coinciding with +a declared universe — is *one* universe: realizations and cache keys attach to +the selection, not to any name. Generated universes are deliberately +**anonymous**: they carry no spec-defined id and are not individually +addressable. To pin or enumerate a specific grid point, *declare* it — with +`base` that is a three-line diff — which also puts a name and a description in +the record. How runners spell per-universe storage paths is a tooling +convention, not spec surface: an option-id join (the prototype's +`f5-nmo3-r2-ecl2-ec1`) reads well for narrow spaces, a digest of the canonical +selection stays bounded for wide ones; either is faithful because identity +rests on the selection. Three resolution rules keep this well-defined: @@ -185,14 +182,11 @@ Three resolution rules keep this well-defined: - **The decision space is a tree, not a grid.** A `when` condition can deactivate a decision under some selections; enumeration recurses over *active* decisions only, and an inactive decision is omitted from the - effective selection and the derived id. Points that differ only in inactive - decisions are the same universe. -- **One namespace, no ambiguity.** Universe and multiverse ids share a - namespace, so a `scope-id` resolves without guessing; a collision between - them is invalid. A declared universe id that spells the derived id of a - *different* selection is likewise invalid (checkable: parse the id as an - option join and compare) — a declared id may only coincide with the derived - id of its own selection, where the two readings agree. + effective selection. Points that differ only in inactive decisions are the + same universe. +- **One namespace, no ambiguity.** Declared universe and multiverse ids share + a namespace, so a `scope-id` resolves without guessing; a collision between + them is invalid. `Multiverse` fields: @@ -200,7 +194,7 @@ Three resolution rules keep this well-defined: |---|---|---:|---| | `id` | `string` | Yes | Multiverse identifier. | | `description` | `string` | No | Human-readable explanation of what the set covers. | -| `universes` | list of universe ids, or `"*"` | One of `universes`/`vary` | Enumerated members (declared or derived ids), or the full valid decision space. | +| `universes` | list of declared universe ids, or `"*"` | One of `universes`/`vary` | Enumerated members, or the full valid decision space. | | `base` | `string` | With partial `vary` | Generator form: declared universe holding the non-varied decisions. | | `vary` | map of `decision_id: "*"` \| option list | One of `universes`/`vary` | Generator form: swept decisions and their option sets. | @@ -214,8 +208,7 @@ multiverse scope: ``` input-reference ::= artifact-id [ "@" scope-id ] -scope-id ::= universe-id | multiverse-id -universe-id ::= declared universe id | derived grid-point id +scope-id ::= universe-id | multiverse-id ; declared ids only ``` `artifact-id` keeps its existing grammar (an Input or sibling Output id, @@ -227,7 +220,7 @@ qualifying it is an error. | Reference | Meaning | |---|---| | `artifact` | Unchanged: the artifact in the *consumer's own* universe. | -| `artifact@` | The artifact materialized under one specific universe — a **pin**. | +| `artifact@` | The artifact materialized under one declared universe — a **pin**. | | `artifact@` | The artifact under *each* universe in the set — a **fan-out**, resolving to a collection. | Semantics: @@ -401,9 +394,8 @@ multiverses: **`astra-tools` (Python CLI + SDK):** - `astra validate` gains checks: universe and multiverse ids are unique in - their shared namespace, and no declared id spells the derived id of a - different selection; every `@` target resolves (declared or derived - universe, or multiverse) and qualifies a reference to an Output; `base` + their shared namespace; every `@` target resolves to a declared universe or + multiverse and qualifies a reference to an Output; `base` chains resolve and are acyclic; a universe without `base` selects an option for every *active* decision; effective selections select valid options and honor `requires` / `incompatible_with`; `vary` names existing decisions and @@ -412,9 +404,10 @@ multiverses: reference targets an output that is active in the pinned universe; a universe-invariant output's recipe uses no `{decisions.}` placeholders. - Runner semantics: cross-universe edges change scheduling — one output can now - demand materialization of an upstream artifact under many universes — and the - derived-id scheme for `"*"` grid points must match the spec so artifact paths - are stable across runners. + demand materialization of an upstream artifact under many universes. Runners + key storage and caching on canonical effective selections; the path spelling + (option join vs. selection digest) is a runner convention, recorded in its + run manifest. **`astra-typescript` (`@astra-spec/sdk`):** regenerate types for the new slots and the `Multiverse` class; the reference grammar change surfaces wherever the @@ -454,11 +447,17 @@ the draft PR resolves them. cache; the alternative — collapsing to distinct realizations — is equivalent to fanning out over the projection of the multiverse onto the output's `decisions`, and could later be expressed explicitly if needed. -- **Derived universe ids — open.** The proposed scheme (option ids joined by - hyphens in decision-declaration order) is simple and collision-free, but ties - ids to declaration order and can get long for wide decision spaces. Is - declaration-order dependence acceptable, or should the id embed decision ids - (`fertility_assessment=f5,...`) at the cost of length? +- **Should generated universes have addressable derived ids? — resolved: no.** + An earlier draft gave every grid point a spec-defined id (option ids joined + in decision-declaration order) usable in pins and enumerations. Rejected: + for a `"*"` over a wide decision space the join grows unboundedly (dozens of + decisions → ids of hundreds of characters, leaking into every artifact + path), it depends on declaration order, and making the spelling addressable + forced extra rules (declared-vs-derived collisions, shadowing). Instead, + identity attaches to the *effective selection*; generated universes are + anonymous; addressing a specific point means declaring it (a three-line + `base` diff, which also names and describes it in the record); and storage + spelling — join for narrow spaces, digest for wide — is a runner convention. - **Should decision defaults define an implicit `baseline` universe? — resolved: no.** An earlier draft derived an implicit, spec-reserved `baseline` universe from `Decision.default` and made it the default base for From d9071b8bf8904070ccf4121f87884f73982778ed Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Wed, 5 Aug 2026 15:00:11 +0200 Subject: [PATCH 08/12] RFC-0003: spell out the late-binding evaluation model for plain references Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index d0b1baca..90d8feef 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -219,10 +219,21 @@ qualifying it is an error. | Reference | Meaning | |---|---| -| `artifact` | Unchanged: the artifact in the *consumer's own* universe. | -| `artifact@` | The artifact materialized under one declared universe — a **pin**. | +| `artifact` | Unchanged: the artifact realized under the *same universe as the consumer* (late-bound; see below). | +| `artifact@` | The artifact realized under one declared universe — a **pin**. | | `artifact@` | The artifact under *each* universe in the set — a **fan-out**, resolving to a collection. | +**Evaluation model.** No output ever names its own universe — the document +stays generic, exactly as today. A universe-scoped output denotes a *family* +of realizations indexed by universe; the index is bound at run time by +whatever demands the artifact (a runner materializing a chosen universe, or a +pin/fan-out demanding specific members). A plain reference is a *bound +variable*: "whatever universe this realization is being built under, consume +the upstream artifact under the same one." A qualifier replaces that bound +variable with a constant (pin) or quantifies it over a set (fan-out). A +universe-invariant output has no index to bind, which is exactly why all of +its references must be qualified or universe-invariant themselves. + Semantics: - **Fan-outs respect activation.** `artifact@` resolves to the From d8e3848e646fa7542aad8c19e74cb82ff1994760 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Wed, 5 Aug 2026 15:09:14 +0200 Subject: [PATCH 09/12] adding minor edits to multiverse design --- rfcs/0003-multiverse-analyses.md | 93 -------------------------------- 1 file changed, 93 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index 90d8feef..a1f1232a 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -223,17 +223,6 @@ qualifying it is an error. | `artifact@` | The artifact realized under one declared universe — a **pin**. | | `artifact@` | The artifact under *each* universe in the set — a **fan-out**, resolving to a collection. | -**Evaluation model.** No output ever names its own universe — the document -stays generic, exactly as today. A universe-scoped output denotes a *family* -of realizations indexed by universe; the index is bound at run time by -whatever demands the artifact (a runner materializing a chosen universe, or a -pin/fan-out demanding specific members). A plain reference is a *bound -variable*: "whatever universe this realization is being built under, consume -the upstream artifact under the same one." A qualifier replaces that bound -variable with a constant (pin) or quantifies it over a set (fan-out). A -universe-invariant output has no index to bind, which is exactly why all of -its references must be qualified or universe-invariant themselves. - Semantics: - **Fan-outs respect activation.** `artifact@` resolves to the @@ -377,66 +366,6 @@ multiverses: scaling: "*" ``` -## Implementation implications & migration - -**`astra-spec` (this repo) — schema, datamodel, docs:** - -- `src/astra/schema/analysis.yaml`: add optional `universes` and `multiverses` - slots to `Analysis` (the `universe` schema is already imported); extend the - `Output.inputs` documentation and validation pattern to admit the `@` - qualifier; clarify in the `Decision.default` doc-string that the default is a - presentational/scaffolding hint with no role in universe construction (its - current "for baseline universes" wording suggests otherwise). -- `src/astra/schema/universe.yaml`: add the `base` slot to `Universe` and - document `decisions` as overrides relative to the base; add a `Multiverse` - class (`id`, `description`, `universes`, `base`, `vary`) with a rule making - `universes` and `vary` mutually exclusive. Revisit the - `UniverseNode.universe` doc-string, which currently hard-codes the - "sub-analysis's `universes/` directory." -- Generated artifacts: `just gen-python`, `just gen-doc`; the published JSON - Schema at `astra-spec.org//schema/…` shifts, which is the propagation - point for both SDKs. -- Docs: `specification.md` (the *Universes* section currently states universes - are "stored separately from `astra.yaml`"; add the reference grammar and a - *Multiverses* section), `index.md` at-a-glance example if touched, `cli.md` - validation rules, `README.md`, and the `examples/` projects. A Steegen-style - example project would exercise the full mechanism in-tree. - -**`astra-tools` (Python CLI + SDK):** - -- `astra validate` gains checks: universe and multiverse ids are unique in - their shared namespace; every `@` target resolves to a declared universe or - multiverse and qualifies a reference to an Output; `base` - chains resolve and are acyclic; a universe without `base` selects an option - for every *active* decision; effective selections select valid options and - honor `requires` / `incompatible_with`; `vary` names existing decisions and - options, `base` is present whenever `vary` is partial, and a generator with - no valid members is an error; enumerated multiverse members exist; a pinned - reference targets an output that is active in the pinned universe; a - universe-invariant output's recipe uses no `{decisions.}` placeholders. -- Runner semantics: cross-universe edges change scheduling — one output can now - demand materialization of an upstream artifact under many universes. Runners - key storage and caching on canonical effective selections; the path spelling - (option join vs. selection digest) is a runner convention, recorded in its - run manifest. - -**`astra-typescript` (`@astra-spec/sdk`):** regenerate types for the new slots -and the `Multiverse` class; the reference grammar change surfaces wherever the -SDK parses `Output.inputs`. - -**Compatibility / versioning:** - -- Plain references keep their exact current meaning, and both new slots are - optional, so existing analyses validate unchanged: the schema change is - **additive — a minor bump** under the versioning policy. Allowing partial - `Universe.decisions` is a loosening: complete universe files stay valid as-is - (an empty diff), and partial ones only become *newly* valid. -- The `universes/` directory remains supported as a place for runner-selected - configurations; in-file `universes` are required only when a universe must be - *referenced* from within the document. Whether the directory is eventually - deprecated in favor of in-file definitions is left open (below); this RFC - does not remove anything. - ## Questions or objections Recorded as open unless marked otherwise; discussion on the tracking issue and @@ -448,9 +377,6 @@ the draft PR resolves them. a multiverse analysis that silently depended on which configurations happened to be declared would be fragile. Recorded so the alternative is not silently re-litigated. -- **Should bare `artifact@*` be allowed in `inputs`? — resolved: no.** Every - fan-out goes through a named, described `Multiverse`. The prototype adopted - this voluntarily and the resulting record is better for it. - **One collection entry per universe, or per distinct realization? — open.** When an upstream output ignores some decisions, several member universes share one realization. The proposal delivers one entry per universe (Steegen-style @@ -480,28 +406,9 @@ the draft PR resolves them. partial universes require an explicit `base`; generators require an explicit `base` when `vary` is partial; `Decision.default` stays a presentational hint. Recorded so the convenience argument is not silently re-litigated. -- **Are diffs too implicit? — open.** With `base` chains, a universe's full - selection is no longer visible at its declaration site. The record arguably - *improves* — the relationship between specifications is the datum a stability - comparison rests on — but tooling should make the effective selection one - command away (e.g. `astra universe show `). -- **Sub-analysis scoping — open.** The prototype is a flat analysis. How does - the `@` grammar compose with sub-analyses — can a parent fan out over a - child's decision space, and how do `UniverseNode` selections participate in - `"*"` enumeration? A conservative first cut: `@` references and multiverse - enumeration operate on the current scope's decisions only. - **Combinatorial guardrails — open.** `"*"` can be astronomically large. Should validators warn (or runners require confirmation) above a universe count threshold, or is that purely a tooling concern outside the spec? -- **Intensional multiverses — partially resolved.** The `base`/`vary` generator - covers the sweep and grid cases from the original sketch (vary some decisions, - hold the rest at a named universe). A general *constraint* language — "all - valid universes where `scaling` is not `none`" — is still deferred; it can be - layered on later (e.g. a `where:` clause) without breaking this design. -- **Report scoping (RFC-0002 tie-in) — noted.** RFC-0002 left "can a report - compare across universes?" out of scope. Universe-invariant outputs give - reports a natural way to present multiverse-level results without resolving - that question in general. ## References From 0f920a64435a767dd97f4e0bc663e4739c03f245 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Wed, 5 Aug 2026 15:34:04 +0200 Subject: [PATCH 10/12] RFC-0003: reconcile with the alternative draft from the tracking issue Adopt '#' as the selector sigil (reserving '@' for a future sources RFC), regex multiverse membership, nested multiverses, union of selection and generation, scalar option selectors, and the tree-wide shared namespace. Remove the universes/ directory entirely (breaking; major bump). Keep the no-privileged-universe stance (explicit base, no defaults completion; 'baseline' is a naming convention only) and the '.' hierarchy separator, recording both divergences in Questions. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 246 +++++++++++++++++++++++-------- 1 file changed, 184 insertions(+), 62 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index a1f1232a..e25fc4a8 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -96,8 +96,13 @@ A universe's **effective selection** is its base's effective selection with the local `decisions` applied on top. The diff form is not just brevity — it is record quality: it states *how two specifications relate*, which is exactly the datum a stability comparison rests on. A universe that spells out every -decision remains valid, so today's complete `universes/` files carry over -unchanged. +decision remains valid; the body of an existing `universes/` file is already a +valid entry, so migration is a move, not a rewrite. + +The spec deliberately privileges no universe (see the design principle above), +but nothing stops a *convention*: authors who want a reference configuration +can simply name a declared universe `baseline` — the name is theirs, not the +spec's. `Universe` fields after this change: @@ -109,56 +114,97 @@ unchanged. | `decisions` | map of `decision_id: option_id` | No | Overrides relative to `base`; must be complete when `base` is absent. | | `analyses` | map of `UniverseNode` | No | Nested selections mirroring sub-analyses; inheritance applies recursively. | -Only in-file universes are addressable: `base` (on universes and on multiverse -generators), enumerations, and `@` scopes resolve against the document's -`universes` — never against `universes/` directory files, which remain valid -for runner-selected configurations (see *Migration*). In-file universes also -join the RFC-0002 tree-path addressing. +**The `universes/` directory is removed.** `astra.yaml` becomes the single +source of truth: keeping the directory as an alternative would demand +precedence rules and invite silent divergence, and external files remain +invisible to the reference grammar (see *Migration*; this is breaking). + +Because `Analysis` is self-similar, `universes` (and `multiverses`, below) may +be declared on any analysis node, and their ids are **unique across the whole +analysis tree** — so a selector resolves tree-wide without qualification. +In-file universes also join the RFC-0002 tree-path addressing. ### 2. Add `multiverses` — named sets of universes, enumerated or generated `Analysis` gains an optional **`multiverses`** slot. A `Multiverse` names a set of universes so cross-universe steps can reference the set by a stable id and readers can see, in one place, which decision space a summary quantifies over. -Membership takes one of three forms: - -- **(a) Enumeration** — `universes:` lists declared universe ids. -- **(b) Generator** — `vary:` maps each swept decision to `"*"` (all of its - options) or to an explicit option list; the members are the Cartesian product - of the swept option sets. Any decision *not* listed in `vary` is held at - **`base`**, a declared universe id — required whenever `vary` does not cover - every decision, so nothing is pinned implicitly. Points excluded by - `requires` / `incompatible_with` constraints are dropped. -- **(c) The full space** — `universes: "*"`: every valid point of the decision - space. Sugar for a generator that varies *every* decision over all of its - options (no `base` — nothing is held fixed). +Membership is declared through two fields — at least one must be present, and +when both are, the membership is their **union**: + +**(a) `universes` — selection.** A list whose entries are, in resolution +order: + +1. an exact declared **universe id**; +2. an exact **multiverse id** — recursively including its members (direct or + indirect membership cycles are invalid); +3. a **regular expression** over declared universe ids (full-string match, + declared universes only — it never matches multiverse ids or generated + members). + +Universe and multiverse ids keep the existing grammar (`^[a-z][a-z0-9_-]*$`), +which contains no regex metacharacters — so an id-shaped entry is always an +exact reference (it must resolve; it is *not* reinterpreted as a regex when +missing), and anything else is a regex. Note the trade the regex form makes: +declaring a new matching universe changes the multiverse's membership in that +revision of the document. + +**(b) `decisions` — generation.** A map from decision references to *option +selectors*: a single option id, a list of option ids, or `"*"` (every option +of that decision). Members are the Cartesian product of the selected option +sets; any decision *not* in the map is held at **`base`**, a declared universe +id — required whenever the map does not cover every decision in scope, so +nothing is pinned implicitly. Combinations excluded by `requires` / +`incompatible_with` are filtered out (unlike a declared universe, where an +invalid selection is an error). As a shorthand for the full space, +`decisions: "*"` (scalar) varies *every* active decision over all of its +options — no `base`, nothing held fixed. + +A decision in a sub-analysis is referenced by its tree path +(`preprocessing.threshold`); an unqualified id is allowed when it resolves in +the current node or names exactly one decision in the subtree, and is an error +otherwise. ```yaml multiverses: - id: model_stability description: Configurations for the model-choice stability comparison. - universes: [rf_standard, svm_standard] # (a) enumeration + universes: [rf_standard, svm_standard] # (a) exact selection + + - id: all_sensitivity + description: Every declared sensitivity configuration. + universes: ["^sensitivity_.*$"] # (a) regex selection - id: scaling_sweep description: Vary feature scaling; all else held at rf_standard. - base: rf_standard # (b) generator: 3 universes - vary: + base: rf_standard # (b) generation: 3 universes + decisions: scaling: "*" - id: seed_scaling_grid description: Scaling x seed grid around the SVM configuration. - base: svm_standard # (b) generator: 2 x 2 grid - vary: + base: svm_standard # (b) generation: 2 x 2 grid + decisions: scaling: [standard, minmax] random_seed: "*" + - id: combined_robustness + description: Named configurations plus the scaling sweep. + universes: [model_stability, "^sensitivity_.*$"] # multiverse + regex + base: rf_standard + decisions: + scaling: "*" # union of (a) and (b) + - id: full_multiverse description: The valid Cartesian product across all decisions. - universes: "*" # (c) the whole valid space + decisions: "*" # the whole valid space ``` -Exactly one of `universes` or `vary` must be present; `base` accompanies `vary` -and is required unless `vary` covers every decision in scope. +`Multiverse.decisions` is `Universe.decisions` generalized from single options +to option selectors, and `base` is the same diff mechanism a `Universe` uses: +a generator is exactly the set of diff-universes +`{base: , decisions: }` for every valid point of the product. A +multiverse expanding to zero universes is invalid. **Identity of generated universes.** A generated member is identified by its *effective selection* — the grid point itself, not the multiverse that produced @@ -185,8 +231,9 @@ Three resolution rules keep this well-defined: effective selection. Points that differ only in inactive decisions are the same universe. - **One namespace, no ambiguity.** Declared universe and multiverse ids share - a namespace, so a `scope-id` resolves without guessing; a collision between - them is invalid. + a single namespace across the whole analysis tree, so a selector resolves + without guessing; a collision — even between declarations in different + nested nodes — is invalid. `Multiverse` fields: @@ -194,43 +241,58 @@ Three resolution rules keep this well-defined: |---|---|---:|---| | `id` | `string` | Yes | Multiverse identifier. | | `description` | `string` | No | Human-readable explanation of what the set covers. | -| `universes` | list of declared universe ids, or `"*"` | One of `universes`/`vary` | Enumerated members, or the full valid decision space. | -| `base` | `string` | With partial `vary` | Generator form: declared universe holding the non-varied decisions. | -| `vary` | map of `decision_id: "*"` \| option list | One of `universes`/`vary` | Generator form: swept decisions and their option sets. | +| `universes` | list of (universe id \| multiverse id \| regex) | At least one of `universes`/`decisions` | Selected members; union with `decisions`. | +| `base` | `string` | With a partial `decisions` map | Declared universe holding the non-varied decisions. | +| `decisions` | map of `decision_ref:` option \| option list \| `"*"` — or scalar `"*"` | At least one of `universes`/`decisions` | Generated members; scalar `"*"` is the full valid space. | -(`"*"` deliberately means the full valid decision space, not "all declared -universes" as sketched in the tracking issue — see *Questions* for the record.) +(`"*"` never means "all declared universes" — as an option selector it means +"every option of this decision," and as `decisions: "*"` it means the full +valid decision space; see *Questions* for the record.) -### 3. Extend the artifact reference grammar with `@` +### 3. Extend the artifact reference grammar with `#` Entries in `Output.inputs` may qualify an artifact reference with a universe or -multiverse scope: +multiverse selector: ``` -input-reference ::= artifact-id [ "@" scope-id ] -scope-id ::= universe-id | multiverse-id ; declared ids only +input-reference ::= artifact-id [ "#" selector-id ] +selector-id ::= universe-id | multiverse-id ; declared ids only ``` +The sigil is `#`, not `@`, following ecosystem convention: `#` selects a +fragment or view *within* a resource (URL fragments, CSS `#id`), while `@` +conventionally pins a *revision* (`actions/checkout@v4`, `pkg@1.2.3`) — and +`@` is deliberately left free for a future sources RFC to address external +analyses at repository revisions, so the two axes compose +(`…output@rev#universe`). One YAML caveat: `#` mid-token is literal +(`score#full_grid` needs no quotes), but a stray space turns the selector into +a comment and the reference into a valid plain `score` — validators should +flag an input whose line-comment looks like a selector. + `artifact-id` keeps its existing grammar (an Input or sibling Output id, -resolving through `from:` chains); `@` cannot appear in element ids today, so -the extension is unambiguous. The qualifier is only valid on references that -resolve to an *Output*: an external `Input` does not vary by universe, so -qualifying it is an error. +resolving through `from:` chains, with `.` as the hierarchy separator +unchanged); `#` cannot appear in element ids, so the extension is unambiguous. +The selector is only valid on references that resolve to an *Output*: an +external `Input` does not vary by universe, so qualifying it is an error. +Whether the selector names a universe or a multiverse is not visible in the +syntax; resolution in the shared namespace determines the cardinality. | Reference | Meaning | |---|---| | `artifact` | Unchanged: the artifact realized under the *same universe as the consumer* (late-bound; see below). | -| `artifact@` | The artifact realized under one declared universe — a **pin**. | -| `artifact@` | The artifact under *each* universe in the set — a **fan-out**, resolving to a collection. | +| `artifact#` | The artifact realized under one declared universe — a **pin**. | +| `artifact#` | The artifact under *each* universe in the set — a **fan-out**, resolving to a collection. | Semantics: -- **Fan-outs respect activation.** `artifact@` resolves to the +- **Fan-outs respect activation.** `artifact#` resolves to the artifact's realizations in the universes of the set where the artifact is *active* — universes excluded by the output's `when` conditions contribute - nothing. (In the prototype, `religiosity_study1_p@full_multiverse` yields 120 + nothing (the multiverse keeps those members; only this artifact's projection + is smaller). A *pin* to a universe where the artifact is inactive is an + error. (In the prototype, `religiosity_study1_p#full_multiverse` yields 120 artifacts while its siblings yield 210.) -- **Universe scoping is inferred, and `@` cuts it.** An output is +- **Universe scoping is inferred, and `#` cuts it.** An output is *universe-scoped* if it declares `decisions`, or if any unqualified input reference resolves to a universe-scoped artifact. Qualified references never propagate scope — the qualifier fixes the universe(s) — and external @@ -244,13 +306,16 @@ Semantics: reference expands to the collection of materialized artifact paths; how the collection is surfaced (space-separated paths, a manifest file, a sidecar) remains the runner's choice, consistent with the existing recipe contract. + The resolved collection must preserve each member's complete effective + selection, so an anonymous generated universe stays scientifically + identifiable without a human-facing id. - **Materialization sharing.** Two universes that differ only in decisions *not* listed on the upstream output's `decisions` share a realization; runners already compute per-output cache keys this way. A fan-out delivers one entry per member universe, with shared realizations deduplicated at the cache, not in the collection (see *Questions*). -Bare `artifact@*` (option 2 in the tracking issue) is **not** proposed: a +Bare `artifact#*` (option 2 in the tracking issue) is **not** proposed: a fan-out must name a declared multiverse. This keeps every cross-universe edge attached to a described, reusable set — one extra stanza in exchange for the summary's domain being part of the record. @@ -296,12 +361,12 @@ outputs: type: figure description: Six-panel histogram of interaction p-values across universes. inputs: - - religiosity_study1_p@full_multiverse # 120 artifacts - - religiosity_study2_p@full_multiverse # 210 artifacts - - fiscal_attitudes_p@full_multiverse - - social_attitudes_p@full_multiverse - - voting_preference_p@full_multiverse - - donation_preference_p@full_multiverse + - religiosity_study1_p#full_multiverse # 120 artifacts + - religiosity_study2_p#full_multiverse # 210 artifacts + - fiscal_attitudes_p#full_multiverse + - social_attitudes_p#full_multiverse + - voting_preference_p#full_multiverse + - donation_preference_p#full_multiverse recipe: command: >- python src/plot_figure1.py --multiverse-results {inputs} @@ -312,7 +377,7 @@ multiverses: description: >- The valid Cartesian product across all five decisions. Option constraints remove NMO1+ECL3 and NMO2+ECL2, leaving 210 universes. - universes: "*" + decisions: "*" ``` **Declared universes, diffs, and a sweep** — the iris example, extended with a @@ -333,8 +398,8 @@ outputs: type: metric description: Macro-F1 under the random-forest vs the SVM specification. inputs: - - f1_score@rf_standard - - f1_score@svm_standard + - f1_score#rf_standard + - f1_score#svm_standard recipe: command: python src/compare.py @@ -342,7 +407,7 @@ outputs: type: metric description: Macro-F1 as a function of feature scaling, all else at rf_standard. inputs: - - f1_score@scaling_sweep # fan-out: one artifact per scaling option + - f1_score#scaling_sweep # fan-out: one artifact per scaling option recipe: command: python src/scaling_sensitivity.py @@ -362,10 +427,42 @@ multiverses: - id: scaling_sweep description: Vary feature scaling; all else held at rf_standard. base: rf_standard - vary: + decisions: scaling: "*" ``` +## Implementation implications & migration + +**Schema (`src/astra/schema/`):** `analysis.yaml` gains inlined `universes` +and `multiverses` slots on `Analysis` (available at every node by +self-similarity) and the `#selector` extension to the `Output.inputs` +reference grammar; `Decision.default`'s doc-string is corrected to a +presentational hint. `universe.yaml` gains `Universe.base`, the `Multiverse` +class (`id`, `description`, `universes` selectors, `base`, `decisions` option +selectors), and an updated `UniverseNode.universe` that points at in-file +universes. Regenerate with `just gen-python` / `just gen-doc`; the published +JSON Schema propagates to both SDKs; docs (`specification.md` *Universes* +section, `elements/`, examples, README) update alongside per the repo's +schema-change ritual. + +**Validation (`astra-tools`):** ids unique in the tree-wide shared namespace; +`base` chains and multiverse membership acyclic; a universe without `base` is +complete over active decisions; effective selections honor `requires` / +`incompatible_with`; regex members are valid full-match patterns over declared +universe ids; every multiverse expands to at least one universe; `#` selectors +resolve and qualify Output references; a pin targets an output active in that +universe; universe-invariant recipes use no `{decisions.}` placeholders; +lint for the YAML `#`-comment footgun. Runner scheduling, cache keys, and +collection serialization are implementation concerns outside this RFC +(identity keys on effective selections). + +**Compatibility:** this is a **breaking** change — a major bump under the +versioning policy. Universe declarations move into `astra.yaml` and the +`universes/` directory is dropped as a supported location; migration is +mechanical (each file's body is already a valid in-file entry). Unqualified +references, the `.` hierarchy separator, and all other existing grammar keep +their exact meaning; the `#` selector itself is additive. + ## Questions or objections Recorded as open unless marked otherwise; discussion on the tracking issue and @@ -404,11 +501,29 @@ the draft PR resolves them. and a spec-blessed default invites treating one path as *the* analysis with the rest as robustness garnish. Consequently: no reserved universe ids; partial universes require an explicit `base`; generators require an explicit - `base` when `vary` is partial; `Decision.default` stays a presentational - hint. Recorded so the convenience argument is not silently re-litigated. + `base` when their `decisions` map is partial; `Decision.default` stays a + presentational hint. Authors who want a reference configuration may *name* a + universe `baseline` by convention — the spec attaches no meaning to the + name. (A contemporaneous alternative draft on the tracking issue completes + omitted decisions from defaults; this RFC deliberately does not.) Recorded + so the convenience argument is not silently re-litigated. - **Combinatorial guardrails — open.** `"*"` can be astronomically large. Should validators warn (or runners require confirmation) above a universe count threshold, or is that purely a tooling concern outside the spec? +- **Selector sigil: `#` over `@` — resolved.** Earlier drafts used `@`. + Adopted `#` following the alternative draft on the tracking issue and + ecosystem convention: `#` selects a view *within* a resource (URL fragments, + CSS `#id`), `@` pins a *revision* (`actions/checkout@v4`, `pkg@1.2.3`), and + `@` stays free for a future sources RFC addressing external analyses at + repository revisions, so the axes compose. Cost accepted: the YAML + stray-space comment footgun, mitigated by a validator lint. +- **Change the hierarchy separator from `.` to `:`? — rejected.** The + alternative draft proposes colon-delimited artifact paths to reserve `.` + for a future artifact-regex RFC. Rejected here: RFC-0002's accepted + tree-path addressing already uses `.`, and breaking every existing + hierarchical reference for a hypothetical future grammar is cost without + present benefit — a future RFC can choose regex delimiters that coexist + with `.` paths. ## References @@ -428,7 +543,14 @@ the draft PR resolves them. - [Snakemake `expand()`](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#the-expand-function) — the workflow-level fan-in pattern over parameter grids. - [`astra-multiverse-example`](https://github.com/anthonyozerov/astra-multiverse-example) - (Anthony Ozerov) — the working prototype this RFC is grounded in. + (Anthony Ozerov) — the working prototype this RFC is grounded in; see also + [`astra-multiverse-example-steegen-2`](https://github.com/anthonyozerov/astra-multiverse-example-steegen-2), + the cross-version specification-search demonstration. +- [Alternative draft](https://github.com/LightconeResearch/astra-spec/issues/52#issuecomment-5145724722) + on the tracking issue — source of the `#` selector, regex membership, + nested multiverses, union membership, and the tree-wide namespace adopted + here; diverges on defaults completion and the `:` separator (see + *Questions*). - Tracking issue: [#52](https://github.com/LightconeResearch/astra-spec/issues/52). - [RFC-0002](0002-decouple-reports.md) — establishes element addressing, which in-file universes and multiverses join as addressable elements. From 0b095dbb3ad6c125d363f70019d7357602716812 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Wed, 5 Aug 2026 15:45:47 +0200 Subject: [PATCH 11/12] RFC-0003: fix breaking-change discrepancy in intro; dedupe rationale into Questions Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01Ah3BGphV5NgkGEhkR3yZDT --- rfcs/0003-multiverse-analyses.md | 76 +++++++------------------------- 1 file changed, 17 insertions(+), 59 deletions(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index e25fc4a8..2a6d70c9 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -42,8 +42,8 @@ In plain language: **universes become addressable elements of the analysis document, declared in full or as diffs of one another; sets of universes get names and constructors; and an output's inputs may reference an artifact *at* a universe or *across* a set of universes.** A plain reference keeps today's -meaning — same universe as the consumer — so existing analyses are untouched. -The proposal has three parts. +meaning — same universe as the consumer — so existing references read +unchanged. The proposal has three parts. One design principle runs through all three: **no universe is privileged by the spec.** The premise of multiverse analysis — and of the stability principle in @@ -96,13 +96,8 @@ A universe's **effective selection** is its base's effective selection with the local `decisions` applied on top. The diff form is not just brevity — it is record quality: it states *how two specifications relate*, which is exactly the datum a stability comparison rests on. A universe that spells out every -decision remains valid; the body of an existing `universes/` file is already a -valid entry, so migration is a move, not a rewrite. - -The spec deliberately privileges no universe (see the design principle above), -but nothing stops a *convention*: authors who want a reference configuration -can simply name a declared universe `baseline` — the name is theirs, not the -spec's. +decision remains valid. Nothing stops the *convention* of naming a reference +configuration `baseline` — the name carries no spec meaning. `Universe` fields after this change: @@ -117,7 +112,8 @@ spec's. **The `universes/` directory is removed.** `astra.yaml` becomes the single source of truth: keeping the directory as an alternative would demand precedence rules and invite silent divergence, and external files remain -invisible to the reference grammar (see *Migration*; this is breaking). +invisible to the reference grammar (breaking — see *Implementation +implications & migration*). Because `Analysis` is self-similar, `universes` (and `multiverses`, below) may be declared on any analysis node, and their ids are **unique across the whole @@ -189,7 +185,7 @@ multiverses: random_seed: "*" - id: combined_robustness - description: Named configurations plus the scaling sweep. + description: Model comparison, declared sensitivity universes, and the scaling sweep. universes: [model_stability, "^sensitivity_.*$"] # multiverse + regex base: rf_standard decisions: @@ -215,10 +211,8 @@ the selection, not to any name. Generated universes are deliberately addressable. To pin or enumerate a specific grid point, *declare* it — with `base` that is a three-line diff — which also puts a name and a description in the record. How runners spell per-universe storage paths is a tooling -convention, not spec surface: an option-id join (the prototype's -`f5-nmo3-r2-ecl2-ec1`) reads well for narrow spaces, a digest of the canonical -selection stays bounded for wide ones; either is faithful because identity -rests on the selection. +convention, not spec surface — identity rests on the selection (see +*Questions* for the record). Three resolution rules keep this well-defined: @@ -259,15 +253,14 @@ input-reference ::= artifact-id [ "#" selector-id ] selector-id ::= universe-id | multiverse-id ; declared ids only ``` -The sigil is `#`, not `@`, following ecosystem convention: `#` selects a -fragment or view *within* a resource (URL fragments, CSS `#id`), while `@` -conventionally pins a *revision* (`actions/checkout@v4`, `pkg@1.2.3`) — and -`@` is deliberately left free for a future sources RFC to address external -analyses at repository revisions, so the two axes compose -(`…output@rev#universe`). One YAML caveat: `#` mid-token is literal -(`score#full_grid` needs no quotes), but a stray space turns the selector into -a comment and the reference into a valid plain `score` — validators should -flag an input whose line-comment looks like a selector. +The sigil is `#`, not `@`: `#` conventionally selects a view *within* a +resource, while `@` pins a *revision* and is left free for a future sources +RFC to address external analyses, so the two axes compose +(`…output@rev#universe`; rationale recorded in *Questions*). One YAML caveat: +`#` mid-token is literal (`score#full_grid` needs no quotes), but a stray +space turns the selector into a comment and the reference into a valid plain +`score` — validators should flag an input whose line-comment looks like a +selector. `artifact-id` keeps its existing grammar (an Input or sibling Output id, resolving through `from:` chains, with `.` as the hierarchy separator @@ -433,29 +426,6 @@ multiverses: ## Implementation implications & migration -**Schema (`src/astra/schema/`):** `analysis.yaml` gains inlined `universes` -and `multiverses` slots on `Analysis` (available at every node by -self-similarity) and the `#selector` extension to the `Output.inputs` -reference grammar; `Decision.default`'s doc-string is corrected to a -presentational hint. `universe.yaml` gains `Universe.base`, the `Multiverse` -class (`id`, `description`, `universes` selectors, `base`, `decisions` option -selectors), and an updated `UniverseNode.universe` that points at in-file -universes. Regenerate with `just gen-python` / `just gen-doc`; the published -JSON Schema propagates to both SDKs; docs (`specification.md` *Universes* -section, `elements/`, examples, README) update alongside per the repo's -schema-change ritual. - -**Validation (`astra-tools`):** ids unique in the tree-wide shared namespace; -`base` chains and multiverse membership acyclic; a universe without `base` is -complete over active decisions; effective selections honor `requires` / -`incompatible_with`; regex members are valid full-match patterns over declared -universe ids; every multiverse expands to at least one universe; `#` selectors -resolve and qualify Output references; a pin targets an output active in that -universe; universe-invariant recipes use no `{decisions.}` placeholders; -lint for the YAML `#`-comment footgun. Runner scheduling, cache keys, and -collection serialization are implementation concerns outside this RFC -(identity keys on effective selections). - **Compatibility:** this is a **breaking** change — a major bump under the versioning policy. Universe declarations move into `astra.yaml` and the `universes/` directory is dropped as a supported location; migration is @@ -517,13 +487,6 @@ the draft PR resolves them. `@` stays free for a future sources RFC addressing external analyses at repository revisions, so the axes compose. Cost accepted: the YAML stray-space comment footgun, mitigated by a validator lint. -- **Change the hierarchy separator from `.` to `:`? — rejected.** The - alternative draft proposes colon-delimited artifact paths to reserve `.` - for a future artifact-regex RFC. Rejected here: RFC-0002's accepted - tree-path addressing already uses `.`, and breaking every existing - hierarchical reference for a hypothetical future grammar is cost without - present benefit — a future RFC can choose regex delimiters that coexist - with `.` paths. ## References @@ -546,11 +509,6 @@ the draft PR resolves them. (Anthony Ozerov) — the working prototype this RFC is grounded in; see also [`astra-multiverse-example-steegen-2`](https://github.com/anthonyozerov/astra-multiverse-example-steegen-2), the cross-version specification-search demonstration. -- [Alternative draft](https://github.com/LightconeResearch/astra-spec/issues/52#issuecomment-5145724722) - on the tracking issue — source of the `#` selector, regex membership, - nested multiverses, union membership, and the tree-wide namespace adopted - here; diverges on defaults completion and the `:` separator (see - *Questions*). - Tracking issue: [#52](https://github.com/LightconeResearch/astra-spec/issues/52). - [RFC-0002](0002-decouple-reports.md) — establishes element addressing, which in-file universes and multiverses join as addressable elements. From 7c834f2afe57a7f72e4175524c3e39697ad9dbf5 Mon Sep 17 00:00:00 2001 From: Francois Lanusse Date: Wed, 5 Aug 2026 16:25:37 +0200 Subject: [PATCH 12/12] Change status from Draft to Active --- rfcs/0003-multiverse-analyses.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0003-multiverse-analyses.md b/rfcs/0003-multiverse-analyses.md index 2a6d70c9..56947427 100644 --- a/rfcs/0003-multiverse-analyses.md +++ b/rfcs/0003-multiverse-analyses.md @@ -1,7 +1,7 @@ --- rfc: 0003 title: Multiverse analyses — in-file universes and cross-universe artifact references -status: Draft # Draft | Active | Accepted | Rejected | Superseded +status: Active # Draft | Active | Accepted | Rejected | Superseded authors: - Francois Lanusse (@eiffl) - Anthony Ozerov (@anthonyozerov)