engine: Continue-As-New compaction rediscovers references with a 394-line walker over the proto AST and retains every step whenever it cannot name one; cel-go's partial evaluation names the exact `steps.<id>.<output>` trails an expression touches, dynamic indexes included

[picatz]

Observed behavior

refs.go is the one CEL-reference walker both drivers share (pkg/flowstate/v1/refs.go:9-20): RootedStepRef (:96) pattern-matches select chains rooted at steps, CollectRefsFromExpr (:138) recurses the google.api.expr.v1alpha1.Expr, and retainAllSteps (:344) is the fail-closed answer "for a walk that has lost the ability to say which step an expression references". docs/ARCHITECTURE.md records the tension under "Static reference analysis is conservative by necessity": the walker "cannot see dynamically constructed expressions", so a steps[inputs.name].value or a steps[vars.stage + "_deploy"].status carries every step forward at every Continue-As-New, and CollectNodeRefs (:377) has grown a Truncated arm and a list of edge cases (:14-20: a map-literal key position, has() on a field a continue_on_error step never set, a whole-step reference beside a field reference) each "found the hard way once".

cel-go has had the mechanism this reimplements since before the pin, and it is in v0.31.0: cel.PartialVars(vars, cel.AttributePattern("steps")) marks a root unknown, cel.OptPartialEval evaluates everything else, and the result is a types.Unknown whose GetAttributeTrails (common/types/unknown.go) lists every steps.<id>.<output>... path the expression actually reached — including through a dynamic index whose operand (inputs.name, vars.stage) is known at compaction time, and including has() (unknown propagation through presence tests landed in cel-go v0.29). Env.ResidualAst is the same machinery pointed the other way.

Why now

Compaction runs in workflow code at the Continue-As-New seam, which is the one place a wrong answer is not retried away: under-retaining fails the run at the next read, over-retaining charges history on every segment (the "spec travels with the run" tension). A precise answer for the dynamic cases is a correctness win and a payload win at once, and it removes the reason refs.go has to be extended by hand every time the language grows a shape (the #492 incident recorded at :361-370 is the pattern).

Desired outcome

Reference discovery for compaction is a partial evaluation of each remaining node's expressions with steps unknown and the run's inputs, vars and locals known, collecting attribute trails into the same refs map NeededOutputs reads today. The AST walker remains the fallback whenever partial evaluation returns an error rather than a value or an unknown, so the fail-closed default is unchanged.

Acceptance criteria

• Shared conformance cases (both drivers) in which steps[inputs.name].value and steps[vars.stage + "_deploy"].status compact to exactly one retained step each, where today every step is retained; the existing refs_test.go cases pass unchanged.
• The has(steps.checkout.error) case (Continue-As-New compaction drops a tolerated step whose only later reference is a has()-guarded absent field #176) still keeps the step's key with the field absent.
• Compaction is bounded by Limits like every other evaluation (partial evaluation is an evaluation), and its cost is charged per node rather than per run.
• refs.go's hand-matched edge cases are each covered by a test that runs both the walker and the partial evaluator and asserts they agree, so the walker can be retired only when the corpus says so.

Constraints and dependencies

• Workflow-side determinism holds: partial evaluation is a pure function of the AST, the known values and the pinned interpreter.
• Comprehensions: fix: track unknowns through comprehensions cel-expr/cel-go#1341 ("track unknowns through comprehensions") is open at f8478f1, so an unknown that flows through map/filter/sum/reduce may today collapse to an error or a value rather than a trail. Until it lands, any expression containing a comprehension over steps keeps the walker's answer; the corpus must include that shape so the fallback is exercised, not assumed.
• Check-before-plan (sibling issue) is not a prerequisite — PartialVars works on parsed ASTs — but with absolute attributes the trails are cheaper to compute.
• Callee isolation (CollectNodeRefs, :361-376): a call: body is still not walked, for the reason recorded there.

Open questions

1. Do AsyncJoinTargets and LoopResultsReferenced (the other two callers named at refs.go:361-364) move to the same mechanism in the same change, or only Continue-As-New compaction first?

[picatz]

Relates to #1583, now this issue's sibling under #1562: refs.go's 394-line walker over the proto AST that this issue asks cel-go's partial evaluation to replace is exactly one of the four duplicate walkers #1583 proposes unifying into one Dependencies(wf) graph. Worth deciding together — a partial-evaluation-based rewrite of refs.go should produce the same graph #1583's derivation would read.

---

Generated by Claude Code