feat(pandas): include= and long-format sidecar dataframes

[termoshtt]

Summary

Wave 1.5 of the v3 alpha pandas surface, scoped to include= filtering plus long-format / id-indexed sidecar DataFrames. The kind=Literal[...] consolidation of today's per-kind *_df methods stays out of scope for this PR.

• include= parameter on every wide *_df accessor across Instance / ParametricInstance / Solution / SampleSet. Default ("metadata", "parameters") matches the v2-equivalent shape; include=[] drops both, include=["metadata"] / include=["parameters"] keep just the named family. Unknown values raise ValueError. Implementation post-filters per-row dicts in entries_to_dataframe based on a IncludeFlags derived from the argument — the ToPandasEntry impls themselves are unchanged.

• Property → method conversion for every *_df accessor. v3-alpha breaking change: instance.constraints_df → instance.constraints_df(). The existing 11 in-tree call sites, the one doctest in solution.rs, and the Sphinx book (en + ja) are all updated. removed_reasons_df / *_removed_reasons_df also lose the property syntax for API consistency, even though they don't take include= (no metadata / parameters columns to filter).

• Long-format / id-indexed sidecar DataFrames on Instance / ParametricInstance / Solution / SampleSet, reading directly from the SoA metadata stores:

  • constraint_metadata_df(kind=...) — id-indexed wide; columns name / subscripts / description, index = {kind}_constraint_id.
  • constraint_parameters_df(kind=...) — long format ({kind}_constraint_id, key, value). One row per (id, parameter_key) pair.
  • constraint_provenance_df(kind=...) — long format ({kind}_constraint_id, step, source_kind, source_id). One row per (id, step) in the provenance chain.
  • constraint_removed_reasons_df(kind=...) — long format ({kind}_constraint_id, reason, key, value). One row per (id, parameter_key) pair, plus one row with key / value = NA for ids whose reason has no parameters.
  • variable_metadata_df() — id-indexed wide; index = variable_id.
  • variable_parameters_df() — long format (variable_id, key, value).

  kind accepts "regular" / "indicator" / "one_hot" / "sos1" (default "regular"); unknown values raise ValueError. Index column names are kind-qualified (regular_constraint_id ≠ indicator_constraint_id) so accidental cross-id-space joins surface in df.head() / df.info(). Long-format *_parameters_df / *_removed_reasons_df rows are sorted by (id, key), and empty long-format DataFrames keep their column schema instead of returning a column-less frame.

• Macro refactor of the per-host kind dispatch. Each of the four hosts gets a constraint_kind_collection! macro that collapses the 16-arm ConstraintKind match (4 sidecars × 4 kinds) into 4 macro invocations. Net −354 lines across instance.rs / parametric_instance.rs / solution.rs / sample_set.rs.

• set_parameter_columns sorts keys before emitting parameters.{key} columns, so wide-*_df parameter column order matches the long-format sort and stays process-stable even when the upstream Constraint.set_parameters is given a std::HashMap (whose iteration is randomized per process). The helper is generic over the hasher so it accepts both FnvHashMap (SoA stores) and std::HashMap (v1::Parameter).

• Codex review pass caught three correctness bugs that are now fixed:

  • Solution / SampleSet sidecar accessors no longer double-count removed constraints (EvaluatedCollection.inner() already includes them — chaining .removed_reasons().keys() produced duplicate rows).
  • WithMetadata<EvaluatedDecisionVariable> / WithMetadata<WithSampleIds<SampledDecisionVariable>> now emit parameters.{key} columns, so Solution.decision_variables_df(include=["parameters"]) and the SampleSet equivalent return the right data.
  • WithSampleIds<SampledNamedFunction> emits parameters.{key} columns instead of a single bare parameters string column, so apply_include_filter honours include=[] here too.

• Book updates (docs/en/**/*.md + docs/ja/**/*.md): tutorial / user_guide / release-note pages flip to the method form, {attr} cross-references for *_df switch to {meth}, and a new "Unreleased" entry on release_note/ommx-3.0.md covers the property → method conversion, include=, and the six new sidecar accessors. 2.x release notes are added to nb_execution_excludepatterns (same treatment as 1.x — historical code samples reference removed APIs).

• Proposal doc (METADATA_STORAGE_V3.md): reflects the narrow per-collection metadata accessors landed in feat(metadata-soa): move per-element metadata onto collection-level SoA stores #843, the explicit "no *_collection_mut()" policy, the actual insert_with(id, c, metadata) -> () shape, and now also the Wave 1.5 landing — include= and the sidecar accessors are marked landed (PR feat(pandas): include= and long-format sidecar dataframes #846), with the deferred kind=Literal[...] consolidation, Series accessors, and Attached wrappers spelled out as the remaining Wave 2 work.

Out of scope

• constraints_df(kind=Literal["regular", "indicator", "one_hot", "sos1"]) consolidating today's per-kind *_constraints_df family — separate PR. This PR keeps the per-kind methods as-is.
• pandas.Series[ID -> Object] collection accessors and the Standalone / Attached two-mode wrappers — Wave 2 of METADATA_STORAGE_V3.md.
• NamedFunction SoA migration.

Test plan

• task python:test passes locally with the full PR.
• task python:book:test rebuilds en + ja docs cleanly with all code-cells executing.
• Snapshot-based python/ommx-tests/tests/test_dataframe_include.py — 9 cases covering default vs. empty vs. metadata-only vs. parameters-only on Instance.decision_variables_df, Instance.constraints_df, Solution.decision_variables_df, SampleSet.decision_variables_df, plus an unknown-flag ValueError check. Snapshots written to __snapshots__/test_dataframe_include.ambr.
• Snapshot-based python/ommx-tests/tests/test_dataframe_sidecars.py — 16 cases covering all 6 sidecar accessors (id-indexed shape, long-format shape, kind= dispatch on every constraint family, qualified index names, empty-provenance baseline, provenance after convert_one_hot_to_constraint, removed-reasons after relax_constraint, removed-reasons with parameters after one-hot conversion, Solution / SampleSet parity with Instance) plus 4 regression tests for the Codex fixes (Solution / SampleSet sidecar dedup with a relaxed constraint, Solution / SampleSet decision_variables_df(include=["parameters"]) actually emits the columns). Snapshots in __snapshots__/test_dataframe_sidecars.ambr.
• 5 consecutive pytest runs all pass — confirms determinism after the long-format (id, key) sort and the alphabetised wide parameters.{key} order.
• Two Codex full-PR review passes; the second came back with one MUST-FIX (a JA / EN inconsistency in instance.md already in place before this PR but surfaced by the docs review) and two non-blocking notes that are scoped out (pd.DataFrame(columns=[...]) produces object dtypes for the empty-frame case; historical 2.x release-note prose references *_df without parens — kept as name reference, not invocation).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Updates the Python (PyO3) pandas-facing surface to support include= filtering on wide *_df accessors, converts *_df properties into methods, and adds long-format / id-indexed “sidecar” DataFrame accessors backed directly by the SoA metadata stores.

Changes:

• Add include= (metadata/parameters gating) across wide *_df methods and implement filtering in entries_to_dataframe.
• Convert all *_df accessors from #[getter] properties to methods and update in-tree call sites/docs/tests.
• Add sidecar DataFrame accessors for constraint/variable metadata, parameters, provenance, and removed reasons (with kind= dispatch for constraint families).

Reviewed changes

Copilot reviewed 36 out of 37 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
python/ommx/src/pandas.rs	Adds IncludeFlags, sidecar DataFrame builders, deterministic parameter-column ordering, and entries_to_dataframe include-filtering.
python/ommx/src/instance.rs	Converts wide *_df accessors to methods with include=, and adds sidecar constraint/variable DataFrame accessors.
python/ommx/src/parametric_instance.rs	Converts *_df accessors to methods with include=, and adds sidecar constraint/variable DataFrame accessors.
python/ommx/src/solution.rs	Converts *_df accessors to methods with include=, and adds sidecar constraint/variable DataFrame accessors.
python/ommx/src/sample_set.rs	Converts *_df accessors to methods with include=, and adds sidecar constraint/variable DataFrame accessors.
python/ommx/ommx/_ommx_rust/init.pyi	Updates stubs from properties to methods, adds include= and sidecar accessor signatures/docs.
python/ommx-tests/tests/test_dataframe_include.py	Adds tests covering include= behavior and validation.
python/ommx-tests/tests/test_dataframe_sidecars.py	Adds tests for all sidecar accessors, kind dispatch, schema stability, and regressions.
python/ommx-tests/tests/snapshots/test_dataframe_include.ambr	Snapshot baselines for include= shapes.
python/ommx-tests/tests/snapshots/test_dataframe_sidecars.ambr	Snapshot baselines for sidecar DataFrame shapes.
python/ommx-tests/tests/test_one_hot_sos1_constraints.py	Updates call sites from property access to method calls for *_df.
python/ommx-tests/tests/test_indicator_constraint.py	Updates call sites from property access to method calls for *_df.
docs/conf_base.py	Expands notebook execution exclude patterns for release notes.
docs/ja/user_guide/instance.md	Updates examples to call *_df() methods.
docs/ja/user_guide/parametric_instance.md	Updates examples to call *_df() methods.
docs/ja/user_guide/solution.md	Updates examples to call *_df() methods.
docs/ja/user_guide/sample_set.md	Updates examples to call *_df() methods.
docs/ja/user_guide/special_constraints.md	Updates refs/examples from {attr}/property style to {meth}/method style.
docs/ja/user_guide/capability_model.md	Updates refs from {attr} to {meth} for removed-constraints DataFrame accessors.
docs/ja/tutorial/tsp_sampling_with_openjij_adapter.md	Updates examples to call *_df() methods.
docs/ja/tutorial/switching_adapters.md	Updates examples to call *_df() methods.
docs/ja/tutorial/solve_with_ommx_adapter.md	Updates narrative/examples to call *_df() methods.
docs/ja/tutorial/share_in_ommx_artifact.md	Updates examples to call *_df() methods.
docs/ja/release_note/ommx-3.0.md	Documents breaking change (methods), include=, and new sidecar DataFrames.
docs/en/user_guide/instance.md	Updates examples to call *_df() methods.
docs/en/user_guide/parametric_instance.md	Updates examples to call *_df() methods.
docs/en/user_guide/solution.md	Updates examples to call *_df() methods.
docs/en/user_guide/sample_set.md	Updates examples to call *_df() methods.
docs/en/user_guide/special_constraints.md	Updates refs/examples from {attr}/property style to {meth}/method style.
docs/en/user_guide/capability_model.md	Updates refs from {attr} to {meth} for removed-constraints DataFrame accessors.
docs/en/tutorial/tsp_sampling_with_openjij_adapter.md	Updates examples to call *_df() methods.
docs/en/tutorial/switching_adapters.md	Updates examples to call *_df() methods.
docs/en/tutorial/solve_with_ommx_adapter.md	Updates narrative/examples to call *_df() methods.
docs/en/tutorial/share_in_ommx_artifact.md	Updates examples to call *_df() methods.
docs/en/release_note/ommx-3.0.md	Documents breaking change (methods), include=, and new sidecar DataFrames.
METADATA_STORAGE_V3.md	Updates proposal/status text to reflect landed include= + sidecar DataFrames and follow-ups.