Simplify policy import model: replace entriesAdditions and importsAliases with entry-level references

[thjaeckle]

Summary

Resolves #2423

Replace entriesAdditions and importsAliases with a single references array on policy entries. Each reference is either:

• Import reference ({"import": "policyId", "entry": "label"}) — refers to an entry in an imported policy
• Local reference ({"entry": "label"}) — refers to an entry within the same policy

Local and import references share the same merge semantics (subjects, resources, and namespaces are inherited additively from the referenced entry); only the lookup site differs.

The previously-introduced allowedImportAdditions field is renamed to allowedAdditions (it now governs both local and import references) and gains explicit three-tier semantics:

• absent → no restriction (upgrade-friendly default)
• empty [] → deny all own additions
• list → only the listed kinds (subjects / resources / namespaces) of own additions survive

allowedAdditions is enforced at resolution time as a runtime filter on the referencing entry's own additions. The strictest set across all references on an entry applies. Own additions that violate the filter are silently stripped at enforcement time — the write itself is not rejected. This is intentional: effective permissions depend on the live state of every referenced entry, which can change.

What's new

• EntryReference model + references array on PolicyEntry
• subjects and resources are now optional on a PolicyEntry (an entry can consist of references only)
• REST API:
  • GET/PUT/DELETE /policies/{id}/entries/{label}/references
  • GET/PUT/DELETE /policies/{id}/entries/{label}/allowedAdditions
• New signals (commands / responses / events) for references and allowedAdditions, with full Ditto Protocol adapter round-trip coverage
• Resolution behavior:
  • One-level resolution: local refs do not chain transitively; cross-policy reach requires transitiveImports
  • importable: never skips and rejects both local and import refs uniformly (the policy author's "do-not-inherit-from-me" signal)
  • Multiple import references from different policies resolve independently, then merge against the strictest allowedAdditions
  • Mutual local references (A → B, B → A) terminate after one pass — each side sees the other's raw content
  • First-listed reference wins on subject ID collisions
  • Hitting MAX_TRANSITIVE_RESOLUTION_DEPTH fails loud (HTTP 400) instead of silent truncation
  • WARN-logging callback fires once per missing reference at enforcement time so silent permission degradation is visible
• Write-time validation:
  • Local refs must target existing entries; import refs must target a declared import and pass READ authorization
  • Reject self-references, duplicate references (same import+entry pair), and local refs to importable: never entries
  • Reject reference modifications that would dangle on import-filter shrinkage (409 PolicyImportReferenceConflictException)
  • PoliciesValidator resolves local references before checking the WRITE-on-policy:/ invariant — a policy that splits the admin subject and the WRITE grant across two mutually-referencing entries is now correctly accepted
• Strict parsing of references and allowedAdditions arrays (non-object / non-string elements rejected with PolicyEntryInvalidException)
• Reject {"import": null} and {"import": ""} explicitly in ImmutableEntryReference.fromJson

What's removed

• entriesAdditions on policy imports (model, signals, strategies, routes, tests)
• importsAliases on policies (model, signals, strategies, routes, tests)
• All matching HTTP routes, Ditto-Protocol adapter mappings, and OpenAPI definitions
• ~12 000 net lines deleted

What's preserved / backward-compatible

• entries filter on import (whitelist of import labels)
• importable (implicit / explicit / never)
• transitiveImports
• Lenient JSON parsing for legacy entriesAdditions / importsAliases fields on snapshot recovery — existing on-disk policies load without an operator migration step (legacy fields are silently dropped from the response)

Documentation

• basic-policy.md rewritten around entry references; adds the unified allowedAdditions semantics, the WRITE-on-references ≡ WRITE-on-entry note, and refreshed power-plant template + 3-level transitive hierarchy examples
• OpenAPI sources updated; ditto-api-2.yml regenerated via the documented npm build
• policy.json JSON schema updated

Test plan

• PolicyImporterTest — 2 / 3-level resolution, NEVER uniformity, NAMESPACES gating, multiple import refs from different policies, conflicting allowedAdditions across imports, empty allowedAdditions strips all own additions, subject-overlap (template wins), mutual local-ref cycle, missing-reference callback, depth-limit failure
• ImmutableEntryReferenceTest — null / empty import rejection
• ImmutablePolicy(Import)Test — legacy field lenient acceptance
• PoliciesValidatorTest — split-admin policy with mutual local refs is valid
• ReferencesValidationStrategyTest — local-ref-to-NEVER write-time rejection; structured-key dedup
• ModifyPolicyImportStrategyTest — shrinking the filter while a reference depends on a removed label returns 409
• ModifyPolicyEntryReferencesStrategyTest — duplicate-ref / self-ref / local-existence / import-existence
• Pre-enforcer: undeclared import target on CreatePolicy / ModifyPolicy rejected; declared import target accepted
• Protocol adapter round-trip for all new and renamed signals
• Policy persistence-actor strategies for the new event types

[hu-ahmed]

Thanks @thjaeckle — the 4 design answers all verified against code and runtime:

• A1 (one-level resolution, no cycles) — confirmed at PolicyImporter.java:307-324
• A2 (409 on delete of referenced entry) — verified live: "policies:entry.referenceconflict" with clear message naming both entries
• A3 (self-reference rejection at write time) — verified on PUT /references, PUT /entries/{label}, and PUT /policies/{id}
• A4 (allowedImportAdditions only for import refs) — confirmed at PolicyImporter.java:366-368

Four follow-up items from live testing against the branch:

1. Local-ref-only policies silently lose authorization

This is the one I'd most like to address before merge. Reproduced live:

Same policy structure; only difference is top-level imports{} being empty vs populated.
  imports: {}                      → Alice GET /features/temp → HTTP 404                                                                                                                                                        
  imports: {ns:template-empty: {}} → Alice GET /features/temp → HTTP 200

Root cause: Policy.java:470 gates resolveReferences behind !imports.isEmpty(). A policy that uses only local references — exactly the shape in the power-plant example in this issue's body — never triggers reference resolution. Suggest calling resolveReferences whenever any entry has non-empty references, regardless of whether imports is empty.

2. Status code / exception type inconsistency

The issue body describes rule 1 as "400 Bad Request." Actual behavior:

Scenario	Status	Error code
PUT /entries/{label}/references with undeclared import	400	policies:import.invalid
CreatePolicy / ModifyPolicy / ModifyPolicyEntry with invalid reference	403	policies:entry.modificationinvalid
Self-reference (any path)	403	policies:entry.modificationinvalid
Ref to non-existent local entry	403	policies:entry.modificationinvalid

PolicyEntryModificationInvalidException is hardcoded HttpStatus.FORBIDDEN at line 57 with a DEFAULT_DESCRIPTION about WRITE-on-policy:/ — it's the "no-writer-left" exception. Reusing it for reference validation produces a 403 where 400 is more appropriate, and a cross-command inconsistency (400 via one endpoint, 403 via others). Suggest PolicyEntryInvalidException (400) or a reference-specific exception.

3. Silent drop of non-object elements in references arrays

PUT /entries/operator/references                          
Body:     [{"entry":"shared"}, 42, "bogus"]
Response: 204                                                                                                                                                                                                                   
Stored:   [{"entry":"shared"}]

Six call sites using .filter(JsonValue::isObject) — ImmutablePolicyEntry.readReferences, ModifyPolicyEntryReferences.fromJson/setEntity, PolicyEntryReferencesModified.fromJson/setEntity,
RetrievePolicyEntryReferencesResponse, AbstractPolicyMappingStrategies.entryReferencesFrom. Same antipattern class flagged before — worth an explicit validating loop that throws PolicyEntryInvalidException naming the bad element.

4. Cross-command READ authorization for references

PolicyImportsPreEnforcer.apply (lines 94-108) routes only ModifyPolicyEntryReferences to validateReferencesModification. For CreatePolicy, ModifyPolicy, ModifyPolicyEntries, and ModifyPolicyEntry — only top-level imports are READ-checked, per-entry reference targets are not. A caller can add an import reference to a template entry they don't have READ on by going through ModifyPolicyEntry rather than /references. Impact scope: information disclosure at resolution time rather than privilege escalation (import refs don't merge subjects), but a cross-command discipline gap worth closing.

Smaller notes:

• PolicyImporter.rewriteLabel drops references from imported entries (6-param factory) while the merge path preserves them (7-param). Either reject references on non-NEVER entries at write time, or extract a
  cloneEntry helper so no path reconstructs a PolicyEntry with fewer than 7 fields.
• Duplicate references ([{entry:"x"},{entry:"x"}]) are accepted as-is; resolution merges twice.
• No ETag support on /references sub-resource.
• ImmutablePolicyBuilder.setPolicyEntry only puts entryReferences when non-empty — re-setting with an empty list doesn't clear prior value (latent).

[thjaeckle]

System tests passed: https://github.com/eclipse-ditto/ditto/actions/runs/24784871727

[thjaeckle]

Ready FMPOV - System tests passed:
https://github.com/eclipse-ditto/ditto/actions/runs/24784871727

[thjaeckle]

System tests:
https://github.com/eclipse-ditto/ditto/actions/runs/24784871727

[hu-ahmed]

LGTM!
Thanks for that massive work