🌱 Ensure COS phase immutability for referenced object approach#2635
🌱 Ensure COS phase immutability for referenced object approach#2635pedjak wants to merge 1 commit intooperator-framework:mainfrom
Conversation
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Enforces ClusterObjectSet phase immutability when objects are sourced via referenced Secrets by requiring referenced Secrets to be immutable and by detecting Secret content changes using recorded hashes.
Changes:
- Add Secret verification in the COS reconciler (immutable requirement + SHA-256 hash comparison) and block reconciliation when verification fails.
- Persist referenced Secret hashes in
.status.observedObjectContainersand extend CRD/applyconfigurations accordingly. - Add/extend unit + e2e coverage and update docs to reflect the new behavior.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/steps/steps.go | Adds new e2e steps for triggering reconciliation, message fragment matching, and checking observed Secret hashes in status. |
| test/e2e/features/revision.feature | Adds e2e scenarios for “mutable Secret” and “recreated Secret with changed content” blocking behavior. |
| manifests/experimental.yaml | Extends CRD schema with .status.observedObjectContainers. |
| manifests/experimental-e2e.yaml | Extends e2e CRD schema with .status.observedObjectContainers. |
| internal/operator-controller/controllers/resolve_ref_test.go | Updates ref-resolution tests to use immutable Secrets. |
| internal/operator-controller/controllers/clusterobjectset_controller_internal_test.go | Adds unit tests for Secret hashing and referenced-Secret verification. |
| internal/operator-controller/controllers/clusterobjectset_controller.go | Implements referenced Secret verification + hashing and blocks reconciliation on violations. |
| helm/olmv1/base/operator-controller/crd/experimental/olm.operatorframework.io_clusterobjectsets.yaml | Mirrors CRD schema changes for Helm packaging. |
| docs/draft/concepts/large-bundle-support.md | Updates documented behavior/conventions for referenced Secrets (immutability + hash enforcement). |
| applyconfigurations/utils.go | Registers apply-configuration kind for ObservedObjectContainer. |
| applyconfigurations/api/v1/observedobjectcontainer.go | Adds generated apply configuration for the new status type. |
| applyconfigurations/api/v1/clusterobjectsetstatus.go | Adds apply support for .status.observedObjectContainers. |
| api/v1/zz_generated.deepcopy.go | Adds deepcopy support for ObservedObjectContainer and status field. |
| api/v1/clusterobjectset_types.go | Introduces ObservedObjectContainer API type and status field. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| func computeSecretHash(secret *corev1.Secret) string { | ||
| h := sha256.New() | ||
| keys := make([]string, 0, len(secret.Data)) | ||
| for k := range secret.Data { | ||
| keys = append(keys, k) | ||
| } | ||
| sort.Strings(keys) | ||
| for _, k := range keys { | ||
| h.Write([]byte(k)) | ||
| h.Write(secret.Data[k]) | ||
| } | ||
| return fmt.Sprintf("%x", h.Sum(nil)) | ||
| } |
There was a problem hiding this comment.
computeSecretHash is vulnerable to ambiguous concatenation (encoding collisions). Different .data maps can produce the same byte stream before hashing (e.g., {"a":"bc"} vs {"ab":"c"}), which can bypass the “content changed” detection without requiring a SHA-256 collision. Use an unambiguous encoding for each entry (e.g., length-prefix key/value, or a 0-byte separator plus length, or canonical serialization of a structured representation) before hashing.
| if len(cos.Status.ObservedObjectContainers) == 0 && len(resolvedHashes) > 0 { | ||
| cos.Status.ObservedObjectContainers = resolvedHashes |
There was a problem hiding this comment.
Hashes are only persisted when .status.observedObjectContainers is empty. Because verifyReferencedSecrets explicitly skips NotFound Secrets, the controller can persist a partial set of hashes on the first pass (for Secrets that happened to exist) and then never record hashes for Secrets that appear later (since status is no longer empty). This breaks the tamper-detection guarantee for late-created referenced Secrets. Fix by either (mandatory): (a) only persisting hashes once all referenced Secrets were found and verified, or (b) merging/upserting newly resolved hashes into status whenever an entry is missing (and persisting the updated list).
| if len(cos.Status.ObservedObjectContainers) == 0 && len(resolvedHashes) > 0 { | |
| cos.Status.ObservedObjectContainers = resolvedHashes | |
| for _, resolvedHash := range resolvedHashes { | |
| found := false | |
| for _, observedHash := range cos.Status.ObservedObjectContainers { | |
| if equality.Semantic.DeepEqual(observedHash, resolvedHash) { | |
| found = true | |
| break | |
| } | |
| } | |
| if !found { | |
| cos.Status.ObservedObjectContainers = append(cos.Status.ObservedObjectContainers, resolvedHash) | |
| } |
| func ClusterObjectSetReportsConditionWithMessageFragment(ctx context.Context, revisionName, conditionType, conditionStatus, conditionReason string, msgFragment *godog.DocString) error { | ||
| msgCmp := alwaysMatch | ||
| if msgFragment != nil { | ||
| expectedMsgFragment := substituteScenarioVars(strings.Join(strings.Fields(msgFragment.Content), " "), scenarioCtx(ctx)) | ||
| msgCmp = func(actualMsg string) bool { | ||
| return strings.Contains(actualMsg, expectedMsgFragment) | ||
| } | ||
| } | ||
| return waitForCondition(ctx, "clusterobjectset", substituteScenarioVars(revisionName, scenarioCtx(ctx)), conditionType, conditionStatus, &conditionReason, msgCmp) | ||
| } |
There was a problem hiding this comment.
This step normalizes whitespace in the expected fragment but not in the actual condition message; if the controller formats messages with newlines/multiple spaces (common when wrapping), strings.Contains may fail unexpectedly. Consider normalizing actualMsg with the same strings.Fields + Join approach before Contains to reduce e2e flakiness while still matching by fragment.
pedjak
changed the title
ClusterObjectSet phases are immutable by design, but when objects are stored in external Secrets via refs, the Secret content could be changed by deleting and recreating the Secret with the same name. This enforces phase immutability for the referenced object approach by verifying that referenced Secrets are marked immutable and that their content hashes have not changed since first resolution. On first successful reconciliation, SHA-256 hashes of referenced Secret data are recorded in .status.observedObjectContainers. On subsequent reconciles, the hashes are compared and reconciliation is blocked with Progressing=False/Reason=Blocked if any mismatch is detected. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pedjak
requested review from
camilamacedo86 and
dtfranz
and removed request for
oceanc80
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
internal/operator-controller/controllers/clusterobjectset_controller.go:1
- Treating
Progressing=False, Reason=Blockedas terminal at the predicate level can prevent the controller from ever reconciling again, even if the user fixes the root cause (e.g., recreates the Secret as immutable, or changes COS spec/generation). If “Blocked” is intended to be recoverable, remove it from this predicate or gate suppression more narrowly (e.g., only suppress when generation hasn’t changed, while still allowing spec updates / explicit triggers to enqueue reconciles).
//go:build !standard
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| resolvedHashes, err := c.verifyReferencedSecrets(ctx, cos) | ||
| if err != nil { | ||
| var sce *secretContentChangedError | ||
| if errors.As(err, &sce) { | ||
| l.Error(err, "referenced Secret content changed, blocking reconciliation") | ||
| } else { | ||
| l.Error(err, "referenced Secret verification failed, blocking reconciliation") | ||
| } | ||
| markAsNotProgressing(cos, ocv1.ClusterObjectSetReasonBlocked, err.Error()) | ||
| return ctrl.Result{}, nil | ||
| } | ||
| cos.Status.ObservedObjectContainers = mergeObservedObjectContainers( | ||
| cos.Status.ObservedObjectContainers, resolvedHashes, | ||
| ) | ||
|
|
||
| phases, opts, err := c.buildBoxcutterPhases(ctx, cos) |
There was a problem hiding this comment.
The PR description/docs say hashes are recorded “on first successful resolution”, but this writes ObservedObjectContainers immediately after verifyReferencedSecrets, before ref resolution/build/apply succeeds. This can record a hash even when later reconciliation fails (e.g., referenced key missing / invalid object), which then makes a later Secret recreation (the only way to fix immutable Secrets) look like “tampering” and permanently blocks reconciliation. Consider persisting hashes only after successfully resolving the referenced objects (e.g., in/after resolveObjectRef when the referenced key is actually used and decoded), or only committing the merge when the reconcile reaches a “Succeeded” point.
| // .Data field. JSON serialization is used to produce a canonical encoding | ||
| // with sorted map keys. | ||
| func computeSecretHash(secret *corev1.Secret) string { | ||
| data, _ := json.Marshal(secret.Data) |
There was a problem hiding this comment.
json.Marshal’s error is discarded. Even if this is unlikely to fail for map[string][]byte, it’s better to handle the error explicitly (e.g., return a sentinel hash + bubble the error up, or fail fast) so unexpected marshal failures don’t silently degrade to hashing nil/partial data.
| data, _ := json.Marshal(secret.Data) | |
| data, err := json.Marshal(secret.Data) | |
| if err != nil { | |
| panic(fmt.Sprintf("failed to marshal Secret data for hashing: %v", err)) | |
| } |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2635 +/- ##
==========================================
+ Coverage 68.95% 69.28% +0.32%
==========================================
Files 139 140 +1
Lines 9891 9994 +103
==========================================
+ Hits 6820 6924 +104
+ Misses 2562 2555 -7
- Partials 509 515 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| // +listType=map | ||
| // +listMapKey=name | ||
| // +optional | ||
| ObservedObjectContainers []ObservedObjectContainer `json:"observedObjectContainers,omitempty"` |
There was a problem hiding this comment.
Instead of requiring the individual objects to never change (and keeping track of per-object hashes), WDYT about computing a single hash of the resulting phases content. If that remains stable, do we care if the underlying organization of the secrets changed?
If we do that, it also means we abstract away any details of how we ultimately end up with those phases (i.e. we don't care if it is inline vs from secrets vs from a bundle ref, etc.), which would be more futureproof.
|
|
||
| func (c *ClusterObjectSetReconciler) SetupWithManager(mgr ctrl.Manager) error { | ||
| skipProgressDeadlineExceededPredicate := predicate.Funcs{ | ||
| skipTerminallyBlockedPredicate := predicate.Funcs{ |
There was a problem hiding this comment.
A terminally blocked COS could become unblocked if the user puts back the expected content though, right? We should continue reconciling to check if the expected content is back in place.
Description
ClusterObjectSet phases are immutable by design, but when objects are stored in
external Secrets via refs, the Secret content could be changed by deleting and
recreating the Secret with the same name. This enforces phase immutability for
the referenced object approach by:
immutable: trueset.status.observedObjectContainerson firstsuccessful resolution
Progressing=False, Reason=Blocked) if any referencedSecret is mutable or its content hash has changed
Reviewer Checklist