feat(metadata-soa): move per-element metadata onto collection-level SoA stores

[termoshtt]

Summary

Moves per-constraint and per-variable auxiliary metadata (name, subscripts, parameters, description, provenance) off per-element structs and onto Struct-of-Arrays stores at the collection / Instance layer (METADATA_STORAGE_V3.md). The Rust SDK and the PyO3 bindings are both fully migrated; cargo test and task python:test pass end-to-end.

This PR is the runtime / Python-API prerequisite for #841 (special-constraint proto v3). The proto-schema decision in #841 — ConstraintMetadata inline per message vs. top-level columnar map — needed the runtime / Python-API direction settled first; that's what this PR establishes.

End state

Rust SDK

• New ConstraintMetadataStore<ID> (generic over the four constraint ID types: regular / indicator / one-hot / SOS1) and VariableMetadataStore under rust/ommx/src/{constraint,decision_variable}/metadata_store.rs. Sparse FnvHashMap-per-field representation with per-field borrowing getters (name(id) -> Option<&str>, subscripts(id) -> &[i64], parameters(id) -> &FnvHashMap<…>, …) and a shared empty-sentinel for the collection-shaped fields. Owned exchange via insert(id, ConstraintMetadata) / remove(id) / collect_for(id).
• ConstraintCollection<T> / EvaluatedCollection<T> / SampledCollection<T> each carry a metadata field with metadata() / metadata_mut() accessors and with_metadata(...) constructors. insert_with(id, c, metadata) performs atomic insert + metadata write.
• Instance and ParametricInstance gain a variable_metadata field with the same accessor pattern, plus narrow per-kind metadata accessors (variable_metadata_mut(), constraint_metadata_mut(), indicator_constraint_metadata_mut(), one_hot_constraint_metadata_mut(), sos1_constraint_metadata_mut() and their immutable siblings) so callers can drain metadata into the SoA stores after builder().build() without exposing invariant-breaking access to the underlying collections.
• The per-element metadata: ConstraintMetadata field is removed from Constraint<S> / IndicatorConstraint<S> / OneHotConstraint<S> / Sos1Constraint<S> / DecisionVariable / EvaluatedDecisionVariable / SampledDecisionVariable.
• Parse / serialize boundary moves to the collection layer. Parse impls now produce (map, ConstraintMetadataStore) / (map, VariableMetadataStore) pairs at the collection level; the collection-level serializers (From<Instance> for v1::Instance, From<ParametricInstance> for v1::ParametricInstance, From<Solution> for v1::Solution, From<SampleSet> for v1::SampleSet) drain the SoA stores and overlay metadata onto each per-element proto via the explicit *_to_v1 helpers.
• The default-metadata From<X> for v1::Y impls (impl From<DecisionVariable> for v1::DecisionVariable and the Evaluated / Sampled / Constraint siblings) are removed entirely. Per-element conversion would have to default every metadata field, which silently drops any caller-supplied metadata; making the helpers pub(crate) and require an explicit metadata: ConstraintMetadata | DecisionVariableMetadata argument forces the silent path to surface as a type error.
• Bare-element bytes round-trips (DecisionVariable::to_bytes / from_bytes and the Constraint / EvaluatedConstraint / SampledConstraint / EvaluatedDecisionVariable / SampledDecisionVariable siblings) are removed. Top-level container to_bytes / from_bytes (Instance, ParametricInstance, Solution, SampleSet, plus the DTOs State, Samples, Parameters) preserve full metadata.
• Evaluate path threads metadata through: Instance::evaluate and Instance::evaluate_samples clone variable_metadata into the produced Solution / SampleSet; SampleSet::get carries it forward into the per-sample Solution along with all four constraint-kind metadata stores. Constraint-side metadata flows through ConstraintCollection::evaluate into EvaluatedCollection / SampledCollection automatically.
• Mutation sites in instance/{slack,sos1,one_hot,log_encode,indicator,evaluate}.rs write through variable_metadata_mut() and the per-kind *_metadata_mut() accessors. Special-constraint promotion paths (one-hot → constraint, indicator → constraint, SOS1 → constraint) carry metadata via insert_with and push_provenance.
• MPS / QPLIB readers populate the SoA stores after Instance::new, replacing the previous per-element metadata writes.
• Memory profile snapshots updated to account for the collection-level visits.

PyO3 wrappers (snapshot model)

• Each wrapper struct now holds its own metadata snapshot:

  Constraint(pub ommx::Constraint, pub ommx::ConstraintMetadata)
  DecisionVariable(pub ommx::DecisionVariable, pub ommx::DecisionVariableMetadata)
  IndicatorConstraint(...), EvaluatedConstraint(...), SampledConstraint(...),
  EvaluatedDecisionVariable(...), SampledDecisionVariable(...)

  Standalone construction starts with Default::default() metadata. Reading from an Instance fills the snapshot from the SoA store via from_parts(inner, metadata.collect_for(id)). Instance.from_components(...) and ParametricInstance.from_components(...) drain each wrapper's metadata back into the instance's SoA stores. Mutations on a wrapper retrieved from an instance therefore do not propagate back; the caller must re-add the constraint / variable to apply changes (matches the prior clone()-based semantics).
• pandas.rs introduces a WithMetadata<'a, T, M> wrapper. ToPandasEntry impls that previously read self.metadata.X now consume WithMetadata<'_, T, ConstraintMetadata | DecisionVariableMetadata>; call sites in Instance / ParametricInstance / Solution / SampleSet pre-snapshot the SoA store and zip the metadata in alongside each item before handing the iterator to entries_to_dataframe.
• from_bytes / to_bytes removed from non-top-level Python wrappers (Linear, Quadratic, Polynomial, Function, DecisionVariable, EvaluatedDecisionVariable, SampledDecisionVariable, NamedFunction, EvaluatedNamedFunction, SampledNamedFunction, Parameter). Only the top-level types (Instance, ParametricInstance, Solution, SampleSet) plus the cross-evaluate DTOs (State, Samples, Parameters) keep them. __init__.pyi regenerated via task python:stubgen to match.

The originally-proposed Standalone / Attached two-mode design with Py<Instance> back-references (write-through getters, live shared state across wrappers pointing at the same id) is intentionally not implemented in this PR. The snapshot model preserves the v2 semantics with minimum surface change; the two-mode design lands together with the Series / include= work in the next wave.

Public API surface

The ommx crate's public surface matches main plus the new SoA accessors required by this refactor:

• pub struct ConstraintMetadataStore<ID> and pub struct VariableMetadataStore (returned by the metadata accessors below).
• pub fn variable_metadata() / variable_metadata_mut() / constraint_collection() / constraint_metadata() / constraint_metadata_mut() / indicator_constraint_collection() / indicator_constraint_metadata() / indicator_constraint_metadata_mut() / one_hot_constraint_collection() / one_hot_constraint_metadata() / one_hot_constraint_metadata_mut() / sos1_constraint_collection() / sos1_constraint_metadata() / sos1_constraint_metadata_mut() on Instance and ParametricInstance. The mutable surface is intentionally narrowed to metadata only — metadata is outside the constraint-collection invariants (a sparse ID-keyed store), so &mut is safe; full &mut ConstraintCollection<T> would expose active_mut() / removed_mut() / insert_with() and let callers register constraints that reference unknown variable IDs.
• pub fn metadata() / metadata_mut() / with_metadata() / insert_with() on ConstraintCollection<T> / EvaluatedCollection<T> / SampledCollection<T>.

The *_to_v1 helpers and the parse submodules stay pub(crate). Module visibility (mod constraint, mod decision_variable, mod instance, mod indicator_constraint in lib.rs) is unchanged — git diff origin/main...HEAD -- rust/ommx/src/lib.rs is empty.

v1 wire-format limitations

v1::Solution and v1::SampleSet only have a single evaluated_constraints / constraints field for regular constraints — they have no fields for indicator / one-hot / sos1 evaluated/sampled constraints. The in-memory Rust types carry those four collections separately (with their own metadata stores), but to_bytes / from_bytes are lossy for the three special kinds. This is a pre-existing wire-format limitation (the matching Parse impls have always initialized those collections to Default::default()) and is documented on the From<Solution> for v1::Solution and From<SampleSet> for v1::SampleSet impls. Wire-shape resolution is the subject of #841.

Docs

METADATA_STORAGE_V3.md updated: status header reflects the two-wave landing, each section now carries an explicit (landed) / (deferred) tag, and the breaking-changes list is split between the two waves. The deferred sections are kept verbatim so the follow-up PR has a concrete spec to land against.

Test plan

• cargo test -p ommx --lib — 474 tests passing, including four bytes-round-trip regression tests:
  • instance::parse::tests::test_parametric_instance_roundtrip_preserves_metadata
  • sample_set::tests::test_sample_set_get_preserves_metadata
  • solution::parse::tests::test_solution_roundtrip_preserves_metadata
  • sample_set::parse::tests::test_sample_set_roundtrip_preserves_metadata
• cargo test (workspace, examples included) — clean compile, all green
• task python:test — full Python suite including ommx + adapter tests (highs / pyscipopt / python-mip / openjij) passes
• Snapshot review for the collection-level memory profile (accepted via cargo insta accept --workspace)
• __init__.pyi regenerated via task python:stubgen

Out of scope (follow-up PRs)

• Proposal: v3 special constraints proto schema #841 — special-constraint proto v3: picks the ConstraintMetadata wire shape (inline per message vs. top-level columnar map) on top of the runtime SoA stores landed here.
• Tighten ConstraintCollection<T>::active_mut() / removed_mut() / insert_with() to pub(crate) (or smaller) — these are heavily used inside the crate but should not be on the public API surface either, per the same invariant-safety rationale that motivated narrowing the Instance accessors here.
• Conversion of instance.constraints / decision_variables / *_constraints Python accessors from dict / list to pandas.Series[ID -> Object].
• The *_df API reshape with the include= parameter and the long-format sidecar dfs (constraint_metadata_df / constraint_parameters_df / constraint_provenance_df / constraint_removed_reasons_df / variable_metadata_df / variable_parameters_df).
• Standalone / Attached two-mode wrappers with Py<Instance> back-references (write-through metadata mutation).
• Doc reorganization (METADATA_STORAGE_V3.md → SDK docs).

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

ommx/rust/ommx/src/instance/parse.rs

Lines 234 to 235 in 7f01bdf

	for id in &absorbed_ids {
	constraints.remove(id);

Clear metadata for constraints absorbed by hints

This loop removes hinted regular constraints from constraints but leaves their entries in constraint_metadata, so the collection can keep orphan metadata for IDs that no longer exist. If a caller later reuses one of those IDs (for example via insert_constraint), the new constraint can pick up stale name/subscripts/parameters during lookups and v1 serialization. Remove metadata for each absorbed ID when dropping the constraint here (and in the analogous ParametricInstance parse path).

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".