Skip to content

Eden PR CI doesn't run test.yml against EVE PR-build images #1188

Description

@eriknordmark

Problem

Eden's PR CI doesn't catch a regression class that fires only when an
EVE PR triggers eden-trusted.yml against an evebuild/pr:<N> image
— it can only be detected post-merge.

Concretely, on an eden pull request the only EVE-touching workflow
that runs is eden_setup.yml:
eden setup / start / eve onboard / clean against a handful of
devmodels and one release tag (eve.tag=8.6.0). test.yml is not
invoked on eden PRs at all.
eden.yml runs test.yml on push to
eden master, but only against lfedge/eve:16.6.0 — a release tag, so
the rootfs's embedded eve_version is 16.6.0-kvm-amd64, exactly
matching what $short_version constructs from eve.tag-eve.hv-eve.arch.

The two eve.tag value shapes in eden config produce different
embedded eve_version strings:

eve.tag Image ref Embedded eve_version
16.6.0 (SemVer) lfedge/eve:16.6.0-kvm-amd64 16.6.0-kvm-amd64
5973 (PR number) evebuild/pr:5973-kvm-amd64 0.0.0-pr5973-<short-sha>-kvm-amd64

A test or test-helper change that depends on eve.tag-derived strings
matching the embedded version (e.g. an exact-match --os-version=…,
an ! stdout '<short_version>' assertion) passes against the first
shape and fails against the second. The first shape is the only one
exercised by eden's own CI; the second is only exercised by
lf-edge/eve/.github/workflows/eden-trusted.yml when an EVE PR's
build completes.

Incident (illustrative)

df8752b added a
! stdout '{{ \$short_version }}' assertion after eveimage-remove --os-version={{ \$short_version }} in tests/update_eve_image/testdata/.
It passed every eden CI run (eden PR, eden master push) and merged on
2026-05-18. The next morning, every EVE PR run started failing the
EVE Upgrade jobs with:

```
FAIL: tests/update_eve_image/testdata/revert_eve_image_update.txt:56:
unexpected match for `-kvm-amd64` found in stdout: -kvm-amd64
```

The fix (#1187) is one-line-per-file in the testdata; the structural
issue is that nothing in eden's own CI exercised the PR-build path.

Other affected runs from the same root cause, recognizable by the
same `unexpected match` signature:

Proposed options

(Not mutually exclusive. Listed in roughly increasing implementation effort.)

A. Add a `test.yml` invocation to eden PR CI against a release-tag image

Cheapest: wire `eden_setup.yml` or a sibling workflow to call
`test.yml` on `pull_request` with `eve_image: "lfedge/eve:"`.
This catches breakage in `test.yml` consumers (the testdata scripts,
the test binaries) but does not close the tag-shape gap — same
shape as today's `eden.yml` post-merge run.

B. Add a `test.yml` invocation to eden PR CI against a PR-build image

Pick a stable, long-lived `evebuild/pr:-kvm-amd64` image (or pin a
specific EVE PR by ID), `docker pull` it, and run `test.yml` against
it on every eden PR. Closes the tag-shape gap directly. Tradeoff: the
pin needs occasional refresh as PRs close / images age out.

C. Add a `test.yml` invocation to eden PR CI against a freshly-built EVE master artifact

Periodically (or on every eden PR), fetch the latest successful
`eve-kvm-amd64-generic` artifact from `lf-edge/eve` master CI, load
it, and run `test.yml` against it. EVE master builds embed a similar
non-SemVer shape (e.g. `0.0.0-master-` or whatever `make show-ver`
emits for HEAD), so the assertion class trips identically. Tradeoff:
cross-repo artifact fetch adds complexity and a permissions surface.

D. Inverse cross-repo trigger: eden PRs trigger an EVE-PR-style smoke

When an eden PR opens or updates, dispatch a workflow run on an
upstream-of-record EVE PR (or on EVE master) that runs the eden PR's
`tests/` against an `evebuild/pr` image. Mirrors what `eden-trusted.yml`
does today but inverted. Closes the gap fully but is the heaviest option.

Option A by itself doesn't actually close this class of escape —
flagging that so it doesn't get picked as "the easy fix" without
realizing.

Acceptance criteria

A reproduction of #1187's bug should fail on an eden PR before merge:

  1. Revert update_eve_image: derive remove version from .ver #1187 on a draft PR against eden master.
  2. PR CI runs at least one job that exercises a non-SemVer `eve.tag`
    shape (an `evebuild/pr:` or EVE-master artifact image).
  3. That job runs `TestEdenScripts/update_eve_image_http` (which
    transitively invokes `revert_eve_image_update`).
  4. The job FAILs at `revert_eve_image_update.txt:56` with the
    `unexpected match for '-kvm-amd64'` signature.

If step 4 doesn't fail, the gap isn't closed.

Related

  • PR update_eve_image: derive remove version from .ver #1187 (the fix this issue is the structural follow-up to)
  • `lf-edge/eve/.github/workflows/eden-trusted.yml` (the only path
    that currently exercises `evebuild/pr:`)
  • `lf-edge/eden/.github/workflows/test.yml` (reusable workflow that
    needs to be invoked on eden PRs to close part of this)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions