feat(numscript): immutable append-only versioned library (EN-1288)

[Azorlogh]

Summary

Rework the numscript library into an immutable, append-only versioned store, per the EN-1288 product-model decision, and bring both of its persisted projections under checker verification (invariant #8).

This supersedes the earlier alpha model (mutable latest slot + per-numscript soft delete/restore + derived tombstone status). Per-numscript deletion, restore, and tombstones are removed entirely — there is no compatibility or migration layer.

Model

• Every version is an explicit full semver (major.minor.patch). Saving with an empty, "latest", or partial version is rejected with NUMSCRIPT_INVALID_VERSION.
• Content entries are immutable. Re-saving an existing (ledger, name, version) returns NUMSCRIPT_VERSION_ALREADY_EXISTS. Changed content or a rollback is published under a new version.
• Each name has a latest pointer equal to the greatest stored semver. A save advances it to max(current, saved); versions may be saved out of order.
• No delete, no restore, no tombstones. DeleteLedger still cascades and removes all numscript data scoped to the ledger.

What's included

1. SaveNumscript — the only write; explicit-semver, append-only (processSaveNumscript).
2. GetNumscript — version selector: ""/"latest" → greatest semver; exact semver → that version; partial ("1", "1.0") → highest match (read-only).
3. ListNumscripts — the greatest version of each named script.
4. ListNumscriptVersions — the latest_version pointer plus every stored version (highest semver first).
5. Reference resolution keeps the "latest"/exact selector in the audited order; admission plans the reads (declares the SubAttrNumscriptVersion pointer + the discovered greatest's SubAttrNumscriptContent coverage) without mutating the order, and the FSM resolves "latest" at apply via the pointer, guarded by a coverage check that yields ErrStaleProposal (retryable) on cross-proposal skew.
6. Checker compareNumscripts (invariant feat(CI): Add GoReleaser #8) — re-derives SubAttrNumscriptContent (full NumscriptInfo compare) and SubAttrNumscriptVersion (greatest-semver pointer) from SavedNumscript / DeleteLedger logs; baseline-seeded under archiving so a post-archive out-of-order save cannot produce a false NUMSCRIPT_MISMATCH.
7. Backup rebuild maintains the greatest-semver pointer, seeded from the checkpoint store so an incremental delta save cannot regress it.

Full parity across gRPC, HTTP REST, and the ledgerctl CLI (numscripts save --version, get, list, versions), plus OpenAPI, api-comparison, cli.md, the numscript-library architecture doc, the checker doc, and AGENTS.md invariant 8.

Design decisions

• latest is a pure projection, always the greatest stored semver — never an independent writable slot. Resolving it on the FSM apply path (which cannot read Pebble, invariant 3) is split: admission reads Pebble to plan/declare coverage without mutating the audited order, and the deterministic FSM resolves the pointer from the cache, falling back to a retryable ErrStaleProposal when a concurrent proposal advanced the pointer between admission and apply.
• Save requires an explicit full semver — no implicit "latest" write. Immutability is enforced at apply against the preloaded content key.

Testing

• Processor, admission (no-order-mutation + greatest-wins overlay), query (version history + resolution), HTTP/gRPC handler, checker (compareNumscripts incl. baseline seeding + non-content-field tamper), state, and backup unit tests — all pass.
• e2e lifecycle tests in tests/e2e/business/numscript_library_test.go (save → list → versions → immutability conflict → reference resolution) — compiled under -tags e2e, left for CI.
• just pre-commit clean (proto/dashboards/CRDs regenerated, go mod tidy clean, golangci-lint 0 issues in main + operator). e2e / scenario / antithesis-workload builds compile.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 2fd312cf-4af3-4c2b-95ef-f84eed2bdf89

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/numscript-versions-restore

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[NumaryBot]

✅ Approve — automated review

The PR introduces an immutable append-only versioned numscript library. All previously identified blockers and major issues have been resolved per the PR discussion record: checkpoint baseline seeding, CheckStore expected-latest baseline seeding, tagged build compile failures, restore log identity sourcing, full-field projection comparison, enum serialisation on the versions endpoint, idempotency key forwarding, restore replay content overwrite, the CLI --version flag validation, and the invariant-8 CheckStore gap are all confirmed fixed. The single remaining open item flagged earlier (silent drop of deleted numscripts in config diff) was noted as a UX gap rather than a correctness blocker and does not warrant blocking the merge. The independent review found no new actionable issues. The PR is ready to merge.

No findings.

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 2 new inline findings.

Summary: #1529 (comment)

[codecov]

Codecov Report

❌ Patch coverage is 54.67033% with 165 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.47%. Comparing base (3818e89) to head (23fde02).
⚠️ Report is 1 commits behind head on release/v3.0.

Files with missing lines	Patch %	Lines
internal/application/check/checker.go	44.96%	40 Missing and 31 partials ⚠️
internal/infra/backup/rebuild.go	4.54%	21 Missing ⚠️
internal/adapter/grpc/client_bucket.go	0.00%	8 Missing ⚠️
internal/adapter/grpc/server_bucket.go	46.66%	4 Missing and 4 partials ⚠️
internal/application/admission/admission.go	79.48%	4 Missing and 4 partials ⚠️
...nternal/domain/processing/processor_transaction.go	42.85%	5 Missing and 3 partials ⚠️
internal/domain/errors.go	30.00%	6 Missing and 1 partial ⚠️
internal/query/numscript.go	74.07%	4 Missing and 3 partials ⚠️
internal/domain/keys.go	66.66%	3 Missing and 3 partials ⚠️
...l/adapter/http/handlers_list_numscript_versions.go	68.75%	3 Missing and 2 partials ⚠️
... and 5 more

❌ Your patch check has failed because the patch coverage (54.67%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@               Coverage Diff                @@
##           release/v3.0    #1529      +/-   ##
================================================
- Coverage         74.75%   74.47%   -0.29%     
================================================
  Files               430      430              
  Lines             45722    45952     +230     
================================================
+ Hits              34179    34222      +43     
- Misses             8495     8629     +134     
- Partials           3048     3101      +53     

Flag	Coverage Δ
e2e	74.47% <54.67%> (-0.29%)	⬇️
scenario	74.47% <54.67%> (-0.29%)	⬇️
unit	74.47% <54.67%> (-0.29%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[NumaryBot]

NumaryBot posted 2 new inline findings.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[gfyrag]

Review note for current head 678508907fd297ad2de3f867a9b4d684b090de53.

I rechecked this after the rebase. The restore replay and preload issues I had seen on the previous head look fixed, but one invariant-8 gap remains:

CheckStore still does not verify the new numscript projections.

This PR makes numscript library state a persisted, load-bearing projection. The top-level audit/log payloads are SavedNumscriptLog, DeletedNumscriptLog, and RestoredNumscriptLog (misc/proto/common.proto:326). The FSM writes the projections through PutNumscript, DeleteNumscriptLatest, and SetNumscriptLatestVersion in internal/domain/processing/processor_numscript_library.go:78, :109, and :151.

Those projections are then used by reads and admission: latest lookup resolves through ReadNumscriptLatestVersion and content lookup in internal/query/numscript.go:51, listing uses the latest-version pointers at internal/query/numscript.go:78, and version status is derived from the same pointer at internal/query/numscript.go:134.

However, internal/application/check/checker.go does not replay these top-level numscript logs into an expected numscript state. The replay switch only handles create/delete ledger and apply logs around checker.go:345, and the final comparison pass only verifies volumes, metadata, transactions, and indexes around checker.go:537. There is no comparison against SubAttrNumscriptVersion / SubAttrNumscriptContent.

That leaves tampering or projection drift undetected: a store can roll back the latest pointer, resurrect a deleted script, remove a content row, or alter stored content and still pass CheckStore as long as the audit chain itself is intact. Since every persisted projection outside the audit chain must be verified by the checker, I think this PR needs a numscript checker pass that replays saved/deleted/restored logs from the audit/log chain and compares both the immutable content entries and latest/tombstone pointers, with tamper tests for pointer drift, missing/extra content, content mismatch, and delete/restore state.

Focused test I ran on the current head:

GOROOT= go test ./internal/application/check ./internal/domain/processing ./internal/query ./internal/infra/backup -run 'Numscript|Check|Rebuild' -count=1

[NumaryBot]

NumaryBot posted 2 new inline findings.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 2 new inline findings.

Summary: #1529 (comment)

[Azorlogh]

@gfyrag — thanks, you were right; the invariant-8 gap is now closed in 5e6feb0.

CheckStore now runs a compareNumscripts pass: the log walk replays SavedNumscript / DeletedNumscript / RestoredNumscript (plus the DeleteLedger cascade, mirroring deleteLedgerData) into an expected state — a save writes the immutable content entry and repoints latest, a delete blanks the pointer, a restore only repoints. The pass then diffs both stored projections (SubAttrNumscriptContent + SubAttrNumscriptVersion) against that expected state and emits the new CHECK_STORE_ERROR_TYPE_NUMSCRIPT_MISMATCH on altered/missing/extra content or pointer drift (resurrected delete, rolled-back pointer). Archive-orphan and deferred-cleanup tolerances mirror compareIndexes.

Tamper tests cover content mismatch, resurrected delete, pointer drift, and missing/extra content, plus clean save/delete/restore cases. Invariant #8 and the checker docs are updated. Ran your command clean: go test ./internal/application/check ./internal/domain/processing ./internal/query ./internal/infra/backup -run 'Numscript|Check|Rebuild' -count=1.

Follow-up: NumaryBot flagged that the content pass only compared the Content field — extending it to the full NumscriptInfo now.

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[gfyrag]

Re-reviewed current head 8b6f93b.

No remaining actionable findings from my pass. The previous checker projection gap looks fixed: compareNumscripts now rebuilds expected Numscript content and pointer projections from the audit log, compares the full NumscriptInfo, and covers tampering/missing/extra/pointer/tombstone cases in tests. The previous backup and preload concerns also look stale/resolved.

Checks run:

• GOROOT= go test ./internal/application/check -run 'Numscript'
• GOROOT= go test ./internal/domain/processing -run 'Numscript'
• GOROOT= go test ./internal/query ./internal/application/admission ./internal/adapter/http -run 'Numscript|Restore|ListNumscript'
• GOROOT= go test ./internal/adapter/grpc -run 'Numscript|ListNumscripts'
• git diff --check release/v3.0...HEAD

[gfyrag]

Requesting changes because the version model needs an explicit product/design decision before this can merge.

The implementation is technically careful, but it currently conflates two different operations:

• restoring a deleted numscript;
• moving the default pointer of a live numscript to any previously stored version (rollback).

That second behavior is outside EN-1288's required outcome and implicitly turns latest into a mutable deployment channel. The current code also gives latest two meanings: an omitted version follows the stored pointer, while the literal "latest" addresses a separate mutable content slot. As a result, a semver can be reported as ACTIVE while an explicit version=latest resolves different content.

We need to investigate and document a simpler model first. The direction currently under consideration is:

• immutable, explicit semver versions;
• latest means the greatest stored semver;
• deletion state is independent from the latest semver;
• restore clears deletion state and does not select an arbitrary version;
• removal of the mutable literal "latest" content slot is acceptable.

Please pause model-dependent implementation work until that contract is settled. The history/listing work, explicit errors, audit/checker coverage, and backup fixes remain useful, but the restore API, pointer semantics, status model, tests, and docs will need to be realigned.

[gfyrag]

Final product-model decision

We completed the model investigation triggered by the Changes Requested review. The authoritative plan is now in EN-1288.

Immutable library

• Numscripts are append-only within a ledger.
• Every stored (ledger, name, version) entry is immutable.
• Save requires an explicit full numeric semver. Changed content or rollback behavior is published under a new version.
• Any unused semver may be added; the per-name latest projection is always the greatest stored semver.
• Per-numscript deletion and restoration are removed entirely.
• DeleteLedger still removes all Numscript data scoped to the ledger.
• No compatibility or migration layer is required for the superseded alpha model.

latest remains in the audited order

The earlier direction to rewrite executable references to exact versions was incorrect and is superseded.

• NumscriptReference.version is required and accepts an exact full semver or the literal "latest".
• Admission may resolve a reference into side data for script parsing, dependency discovery and preload planning, but it must not mutate the accepted order.
• The FSM resolves "latest" through the existing per-name SubAttrNumscriptVersion attribute, redefined as the greatest stored semver.
• The FSM then loads the exact content through the coverage-gated Scope without writing the resolved version back into the order.
• AuditItem.SerializedOrder must retain version: "latest". A generated result/log may expose the concrete version.
• Executable partial selectors (1, 1.2) are removed for v3.0; read-only APIs may retain them.

No second latest attribute is needed. The existing NumscriptVersionKey -> NumscriptVersionValue.version projection must be clarified/renamed and verified by the checker.

Coordination requirements

The attribute alone is not enough because dependency discovery happens before proposal:

• serialize SaveNumscript and executable latest-reference planning per (ledger, name) so a later admission cannot discover dependencies from an old script while the FSM observes a newer latest pointer;
• plan same-bulk requests sequentially: later saves cannot affect earlier references;
• compare persisted state with prior overlay saves, and never let an overlay lower version hide a greater persisted version.

Do not freeze the admission-selected exact version into the order or an unaudited business-resolution sidecar.

Consequences for this PR

Please remove:

• DeleteNumscript and RestoreNumscript APIs, orders, logs, CLI/actions and tests;
• tombstones, empty latest pointers, deleted / includeDeleted;
• ACTIVE/SUPERSEDED/TOMBSTONED version statuses;
• arbitrary pointer reassignment and live rollback;
• the mutable literal "latest" content slot and its key encoding;
• admission's mutation of NumscriptReference.

Please retain and realign:

• immutable explicit-semver content;
• history listing as latestVersion plus ordered {version, createdAt} entries;
• exact dependency discovery without order mutation;
• FSM-side latest resolution using declared latest/content preloads;
• audit, checker, archive-bound replay and backup/rebuild coverage;
• explicit duplicate/invalid-version errors and focused concurrency tests.

The cross-cutting invariant is: once converted to a raftcmdpb.Order, the accepted business payload is immutable until audit capture. Coverage bits and proposal-level execution metadata remain allowed; state-dependent business rewrites do not.

The Changes Requested review remains in effect until the PR matches this final model.

[gfyrag]

Re-reviewed against the settled product-model decision (EN-1288) — the requested realignment is fully applied: restore API removed, order no longer mutated by admission (FSM resolves "latest"→greatest semver, coverage-gated with a loud fail), derived status model removed (fields reserved), tests and docs realigned to the new contract. The parts I'd flagged as fine are intact (history/listing, explicit errors, backup fixes), and my invariant-#8 gap is closed by the new compareNumscripts checker pass. Build + numscript test packages green.

Two non-blocking notes: (1) the branch is currently DIRTY — please merge release/v3.0 (merge-base conflict only, compiles clean on the head); (2) a few stale comments still describe the old delete/restore/tombstone model (e.g. raft_cmd.proto:324, checker.go:189-192/951-952/971-972/1015) — worth a scrub since the code is now correct. Approving; those don't block.

[NumaryBot]

NumaryBot posted 3 new inline findings.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot review complete: no remaining inline findings.

Resolved 1 stale NumaryBot review thread (1 fixed, 0 outdated).

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot review complete: no remaining inline findings.

Resolved 1 stale NumaryBot review thread (0 fixed, 1 outdated).

Summary: #1529 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1529 (comment)

[NumaryBot]

NumaryBot review complete: no remaining inline findings.

Resolved 1 stale NumaryBot review thread (0 fixed, 1 outdated).

Summary: #1529 (comment)