[WIP] evetest: kvm→k boot-disk conversion tests + volverify app - #6267
Draft
eriknordmark wants to merge 26 commits into
Draft
[WIP] evetest: kvm→k boot-disk conversion tests + volverify app#6267eriknordmark wants to merge 26 commits into
eriknordmark wants to merge 26 commits into
Conversation
Adds a test application that writes a deterministic, self-verifying fill/delete pattern to an application volume and later checks it, to detect corruption caused by a watchdog-interrupted EVE-kvm to EVE-k offline filesystem shrink. It is the volume-content ground truth the resize soak pairs with the on-device fsck marker. Each block's content is a pure function of its logical identity (file id and block index within the file, not physical placement), so a successful shrink relocation verifies clean while a torn extent tree surfaces as a misplaced, zeroed, or torn block. A seeded op stream drives create/delete churn with a crash-safe two-slot committed index, so the writer resumes across reboots and the verifier classifies each file as ok, present-corrupt, orphaned (in lost+found), lost, or resurrected. verify accepts an off-volume committed-index floor so loss of the last work is still detected when fsck clears both the data and the on-volume bookkeeping. Ships as an ubuntu image driven over SSH, with unit tests and a root-gated loopback-ext4 fidelity script. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lets a test attach one or more empty data volumes to an application beyond its image. Each is created as a blank disk of a given size (VCOT_BLANK, no content tree) and mounted at a chosen path. This is needed to exercise a data volume through the EVE-kvm to EVE-k offline shrink: the volume must be a large file in /persist so the resize relocates its blocks. The existing app config only carried the image volume and its size. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a test that checks whether a watchdog-interrupted EVE-kvm to EVE-k offline shrink corrupts an application data volume. It deploys the volverify app with a large blank data volume, fills it before the upgrade, drives the kvm to kubevirt conversion (which repartitions and shrinks /persist, relocating the volume's blocks), and re-verifies the pattern afterward. The watchdog fault is expected to come from the target EVE image rather than the harness, so the test simply upgrades to that build. It fails when the volume comes back present but corrupt (the case EVE would silently serve to the app) and surfaces the recoverable orphaned/lost counts for the soak to tally. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add TestKvmToKResize: it drives EVE-kvm→EVE-k across the offline boot-disk shrink+grow while keeping a container app and its data volume, then asserts the volume's marker survived the conversion. Includes ordered post-conversion readiness gates (volumemgr Initialized, Longhorn StorageClass, app RUNNING, app SSH) for failure localization, a /persist + Longhorn node-disk accounting capture (pre-conversion baseline plus a sampler through the EVE-k startup window) for diagnosing the post-conversion app-provisioning wedge, and DISK_SIZE_MB / RAM_SIZE_MB / CPUS / DATAVOL_MB parameters with floors matching the eden kvm→k bringup (64 GiB disk, 16 GiB RAM, 8 vCPU). Also add EdgeDevice.ExpectAdditionalReboots so a test can declare the extra device-initiated reboots an offline shrink+grow performs, keeping the harness teardown reboot-count check accurate. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Writer.Run now returns the committed high-water mark and treats ENOSPC as a normal end state rather than an error: filling the volume is the expected end for the corruption soak, not a failure. The write CLI prints the committed index, and the app-volume shrink test passes that reported index to `verify --expect-committed` instead of assuming every requested op was written (a volume that fills first commits fewer). Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The app-volume shrink-corruption test was written before the kvm→k conversion had ever completed under evetest, and it carried none of what that took: it provisioned from the installer image (whose ESP holds a 0-byte file the offline grow cannot copy), gave the app a local network instance that never reconverges onto the kubevirt VMI, went straight from the small 16.6.0 base to the kubevirt target without the intermediate hop that lands the conversion code, and left the device on the framework's 4 vCPU / 8 GiB defaults. It would have failed long before reaching a volume. Move it alongside TestKvmToKResize so it runs the sequence that is known to work and shares its assertions and diagnostics, and swap in volverify plus a blank data volume for the plain container app and its string marker. After the conversion the volume is no longer mounted at its MountDir, so the verify locates it among the guest's block devices and mounts it read-only without a journal replay, which would otherwise heal the torn state being measured. A volume that comes back empty is reported as such rather than as a lookup failure — that is what a run sees when the post-resize manifest check removed a torn volume and EVE recreated it — and the manifest state is captured alongside it. The default volume size drops to 256 MiB, below the size at which the app redeploy wedges in a Longhorn CSI create race that would mask the result. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
A fault-injected conversion that converges is indistinguishable from a clean one at the harness level: the device reboots, comes back on the target, and the geometry asserts pass either way. The first Stage C run passed with a spotless volume and left no evidence of whether the stress watchdog had cut the offline resize at all, so the result could not be read as either "the shrink preserves the volume" or "nothing was injected". Capture the resize attempt counter after the conversion, which storage-resize.sh advances once per resize boot and which therefore distinguishes a single clean pass from a re-driven one, together with the watchdog device node and the recorded reboot reasons. The resizer's watchdog exits quietly on a guest that has no watchdog device, which would silently reduce the whole test to a no-fault run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
A device under evetest survives a hardware-watchdog timeout that would reset real hardware. The q35 machine brings the ICH9 LPC bridge, and with it the iTCO watchdog EVE drives through /dev/watchdog, but the bridge only latches a status bit when the timer expires unless its no-reboot property is cleared and an expiry action is set. So the guest runs on as if nothing had happened. That hides any regression in EVE's own watchdog handling, and it silently disarms tests that deliberately fire the watchdog — a fault-injection run then looks indistinguishable from a clean one, which is how a spotless result can be mistaken for evidence that the code under test is sound. Configure the same two QEMU settings the eden-based harness already uses, so the guest behaves like the hardware it stands in for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The harness allows a fixed 20 minutes for a device to come back running an upgraded version. That suits an ordinary base-OS upgrade, but a cross-flavor boot-disk conversion also repartitions the disk offline across several restarts and then brings up a whole container-cluster stack, which takes very nearly the whole budget: two conversion runs on the same image measured 19m46s and just over 20m, so the second was failed as "timed out" while it was still making progress and had in fact already booted the target image. Let a test raise the wait for its own device handle, and do so in the two conversion tests. Without this the pass/fail verdict on a conversion is decided by a couple of minutes of host load rather than by the conversion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The app-volume shrink test interrupts the offline resize by having the resizer arm /dev/watchdog and stop feeding it. When the guest has no such device the resizer exits quietly instead, nothing is interrupted, and the conversion runs to completion — so the volume verifies perfectly and the run reads exactly like evidence that an interrupted shrink preserves application data. That is the worst possible failure mode for a fault-injection test: it produces a reassuring result rather than an error. Assert the device node up front, on the baseline, so a guest that cannot deliver the fault fails in seconds with an explanation. Also correct what the post-conversion evidence capture reads. The resize attempt counter looked like the ideal witness — it advances once per resize attempt — but it is deleted on the success path, so after a completed conversion it always reads empty no matter how many attempts it took. Capture the recorded boot reasons instead, since a watchdog reset is reported as one, and leave a note so it is not reintroduced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
A /dev/watchdog character node can exist with no driver behind it, so testing for the node proves nothing about whether the guest can deliver a watchdog reset — a run observed the node present while wdctl could not read any information about it. Key the check on sysfs instead, where a driver only appears once it has registered. Do not test that the node opens, either: EVE's watchdog service holds it once the system is up, so a busy device is the healthy case here rather than a fault. Note in passing what this check cannot establish, since it runs with everything up whereas the resizer runs from an onboot container long before that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Data volumes above roughly 256 MiB make the app redeploy on EVE-k wedge: a PVC stays Pending forever behind a Longhorn CSI create/verify race. It is stuck rather than slow, so the existing wait can only ever time out, which caps the test at a volume size far too small to be relocated by the shrink in any interesting quantity. Wait in rounds instead and, between them, delete the Pending PVCs so the provisioner re-drives them — the documented recovery for this wedge. Never delete the data volume's PVC or its import scratch. That volume holds the pattern the test verifies and EVE recreates a deleted volume empty, so the verify would find a blank volume and the run would report a clean pass: a false negative that looks like a result. The size comparison is therefore biased towards protecting, and when the wedged PVC is the protected one no recovery is attempted and the run fails honestly. Sizes are used because the harness does not know the volume UUIDs EVE assigns; the classification is logged so a protected-everything round is visible rather than silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The framework's upgrade wait returns only once EVE has committed the new partition, which is a trial period lasting well past the point where the device is already running the target. On a conversion that is ten minutes of a twenty-six minute wait, and none of it is needed here: by then the boot disk has been repartitioned and what remains to be established is whether the application and its volume come back. Return as soon as the target is the booted partition and drive the existing readiness gates instead, so a failure lands on the step that actually failed — the cluster, the storage class, the volume — rather than on a timer that expired while the conversion was progressing normally. A rejected image still surfaces immediately, since the wait fails fast when EVE flags one FAILED. Whether the partition was ever committed is recorded at the end, where it explains a later oddity without gating anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
On a nearly empty /persist the app's data volume is allocated in low blocks, well below the size the shrink cuts the filesystem down to, so the shrink never has to relocate it — and with almost nothing to move the shrink finishes in about a second, too fast for the stress watchdog to interrupt. Runs therefore came back clean for a reason that had nothing to do with whether an interrupted relocation is safe. Fill /persist with incompressible data before deploying the app, so its volume is allocated at the top of the filesystem, then delete the lowest files once the pattern is written. That leaves the volume and a few GiB of survivors above the boundary: enough relocation work for the shrink to run long enough to be cut, while freeing the low blocks the shrink needs to fit and the space EVE-k needs afterwards. The bytes must be incompressible or the qcow2 backing file stores them sparsely and the relocation reads and writes nothing. Then refuse to continue unless the volume really does have blocks above the boundary the resizer reports, since a volume sitting entirely below it is untouched by the shrink and a clean verify would prove nothing. Creating a volume on a deliberately full filesystem also needs EVE's disk check disabled, as the fill is transient and trimmed away before the conversion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The placement assertion looked for TargetBytes where the resizer emits targetBytes, so it could never find the shrink boundary and failed every run before the conversion started. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Neither of these had ever executed, so both were wrong in ways no run had yet exposed. The size comparison that decides whether a Pending PVC may be deleted understood only Gi and Mi suffixes, but kubectl prints these PVCs as a plain byte count. Every size therefore measured as zero and every Pending PVC was a candidate for deletion — including the data volume and its import scratch, the two it exists to protect. Deleting those does not merely lose a recovery: EVE recreates the volume empty, the verify then finds nothing wrong, and the run reports success. Handle each form kubectl may print and protect anything unrecognised, so a parsing gap can only cost a recovery rather than the result. The volume placement check read the physical extent's start column as though it were the end, and would have broken outright had filefrag closed up the space inside its "start.. end" range. Close that space first and take the range's second half, which holds regardless of alignment. Verified against real filefrag output in both spacings and against a file with no extents. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The test asserted that no file came back corrupt but never recorded what a filesystem check made of the same volume, so it could not answer the question it exists to answer: when an interrupted relocation damages application data, would anything routine have noticed? That gap matters because the two verdicts are expected to disagree. A structural check is blind to data blocks relocated wrongly but left self-consistent — on a loopback filesystem, zeroing four mebibytes of data left e2fsck reporting a clean filesystem with an empty lost+found while the content verify found seventy-eight files silently full of zeroes. Either verdict alone is ambiguous; the pair is the finding. Check the volume before anything mounts it, since a mount replays the journal and can repair the damage under measurement, and check read-only so the check itself changes nothing. Locate the device by size rather than by mounting it to look inside. Emit both verdicts as a single line per run, so a soak can accumulate hundreds of them and the disagreement rate becomes the result rather than an anecdote. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The filesystem check taken before the volume is mounted cannot tell real damage from bookkeeping. The volume is never cleanly unmounted, so its journal is unreplayed and the superblock disagrees with the disk by construction: one run reported 235353 free blocks against an actual 4644. That check therefore reports errors every time and, taken alone, would have made the structural column a constant in a soak — useless for spotting the case worth finding, where a check calls a damaged volume clean. Check it a second time once the content verdict is recorded, with the journal replayed and repairs allowed, so anything still reported is genuine. That modifies the filesystem, which is why it runs last and why the volume is unmounted first. Record both readings: the pair separates stale accounting from damage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
A volume captured while it was still being written has stale superblock counters, the orphan-file flag set and extent trees e2fsck would rather rewrite. Repairing those returns a non-zero status on every run, so labelling any non-zero status as damage marked healthy iterations as hits and would have buried the signal a soak exists to find — the same mistake as reading the pre-replay status, one step later. Classify on the findings instead: unattached or deleted inodes, crossed or illegal blocks, bitmap differences, anything reaching lost+found. Two soak iterations that reported damage under the old rule repaired only free-count and orphan-flag bookkeeping, with all file contents verifying clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The app failing to reach RUNNING after the kvm->k conversion is the most common way this test fails, and it was the one failure that produced no diagnostics: the deferred capture of PVC state, csi-provisioner and longhorn-manager logs was registered below the app-RUNNING wait, so a failure inside that wait skipped it entirely. Six consecutive wedges were recorded with nothing to explain them. Register the capture before the wait and report which stage was reached, so a run that never gets the app up still leaves the evidence behind. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Five diagnostic probes globbed /persist/newlog/collect/*.log. That directory holds only the last few minutes of records — newlogd closes a collect file and moves it to the gzipped queues once it passes 550000 bytes or a 300s timer expires — so the probes saw almost nothing of the window they were meant to explain. The glob was narrower still: the chunks are named dev.log.keep.<n> and dev.log.upload.<n>, so *.log matched only current.device.log, a symlink to the live keep stream, missing even the upload half beside it. Every wedge and watchdog probe was reporting a near-empty result that read as "the event never happened". Route them through the retrieval form the eve-device-logs skill already documents, which walks the whole newlog tree one file at a time. Checked against a collected device tree of 133 chunks: the old glob found no BootReason records, the new form finds 85. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
Every check of the migrated data volume ran inside the app guest, so the post-conversion app wedge destroyed the whole verdict — filesystem check and content check alike. Seven consecutive iterations exercised the interrupted shrink and recorded nothing about the volume, including one that cut the shrink twice with the volume 17 GiB inside the relocated range. Read the volume on the device instead, before the cluster can ingest it. upgradeconverter parks the carried-over volumes in /persist/vault/volumes-kvm as plain ext4 images, and the data-volume path leaves the source in place when the PVC rollout fails, so a wedged import is exactly the case where the file is still there. containerd has already unpacked the app image, so the same volverify binary the app would run is on disk and needs no transfer. Verified live: fsck read-only on the image, loop-mounted read-only alongside an in-flight import with no contention, verify clean at committed=2944. The app-side path stays as the cross-check that covers the import itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The device reports itself on the target before upgradeconverter's post-vault phase has necessarily moved the carried-over volumes into volumes-kvm, so sampling for the file immediately after the conversion is a race. Losing it looks like success: a missing file is treated as nothing to verify, and the iteration silently produces no volume verdict at all. One run sampled sixteen seconds after the device came up, skipped, and only then did volumemgr start logging import errors against the very file that was reported absent. Poll for the file for up to ten minutes instead, and report how long the wait took so a slow relocation is visible rather than inferred. Also read clear volumes from clear/volumes, where kvmMigratedSourcePath looks for them; the previous path had them under a -kvm sibling that only encrypted volumes use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: eriknordmark <erik@zededa.com>
The device-side check looked for the volverify binary before entering the wait loop, so a containerd snapshot that was not yet unpacked made it exit early and skip the wait it had just been given. A run that ended up with both artifacts present recorded no volume verdict, reporting the binary as missing. Wait for the binary and the relocated volume in one loop, and report the elapsed wait either way so a slow appearance is visible rather than inferred from a skip. Validated on hardware: four consecutive conversions recorded a complete app-independent verdict with nothing skipped, three of them alongside a working app-side check that agreed with it. Each reported a zero-length wait, so the common case costs nothing; the wait still has not been observed absorbing a slow relocation, which is the case it exists for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A corruption soak only earns a row if the iteration actually examined a volume, and three things stopped that happening. EVE's post-resize check hashes each volume against a pre-resize manifest and deletes any that mismatches, so the torn volumes this test hunts were gone before it could look at them; a marker now has them quarantined instead, and the device-side verify reads the quarantined copy exactly like an intact one while recording that EVE had already condemned it. Finding nothing to verify is now reported instead of passing silently, and a missing filler directory fails the trim rather than continuing. The filler also stops short of the data volume's size and lives under /persist/log: filling to the requested peak and then writing the volume into what little remained drove the device into low-disk maintenance mode and a reboot, and onboot.sh removes /persist/tmp on every boot, voiding the placement the fill exists to create. DEVSIDE_ONLY ends the test once the volume verdict is in, skipping the app-side checks that account for most of a run's wall time. At data-volume sizes where the post-conversion app reliably wedges those checks re-derive a known failure, so a soak collects far more verdicts per day without them — at the cost of the wedge diagnostics, which nothing else produces. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the post-conversion app never gets its volume, the failure diagnostics named the CDI upload controller but never the upload server — the process that receives the transfer, and the one observed Ready and idle while the import never completes. It is captured now, including the previous container's log, because on smaller volumes the upload pod has already terminated by the time this path runs and the live fetch came back empty. Alongside it: namespace warning events, which outlive the pods that caused them and are where an attach failure surfaces; the per-PVC upload progress, which distinguishes a transfer that never started from one that stalled partway; and the Longhorn volume state underneath, since an upload can idle because the volume was never brought up on the uploader's node. The RolloutDiskToPVC window grew because retry noise had been filling it exactly, and the recovery actions moved to their own probe so that noise cannot crowd them out. The persist snapshot also runs the conversion pre-flight now, so a declined conversion comes with the sizes and the policy floor behind its one-line reason rather than only the reason. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6267 +/- ##
==========================================
+ Coverage 22.85% 23.21% +0.36%
==========================================
Files 509 519 +10
Lines 93334 95050 +1716
==========================================
+ Hits 21331 22069 +738
- Misses 70268 71061 +793
- Partials 1735 1920 +185 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
milan-zededa
reviewed
Aug 4, 2026
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| EVETEST_ORG ?= lfedge | ||
| IMAGE = $(EVETEST_ORG)/evetest-volverify |
milan-zededa
reviewed
Aug 4, 2026
| // reboots additional times; the number is known only to the test driving it (a | ||
| // plain kvm<->k conversion with no resize reboots differently than a shrink+grow), | ||
| // so the test declares it here to keep the teardown reboot-count check accurate. | ||
| func (d *EdgeDevice) ExpectAdditionalReboots(n int) { |
Contributor
There was a problem hiding this comment.
Perhaps additional expected reboots and upgrade timeout can be just optional arguments for EdgeDevice.UpgradeEVE()?
milan-zededa
reviewed
Aug 4, 2026
| @@ -0,0 +1,1347 @@ | |||
| // Copyright (c) 2026 Zededa, Inc. | |||
Contributor
There was a problem hiding this comment.
How about moving these tests to tests/upgrade and adding them to TestUpgradeSuite?
Contributor
There was a problem hiding this comment.
Then you could also create package-global definitions of:
evetest.TestParameterDefinition{
Key: initialEVEVersionParamKey,
DefaultValue: "16.0.0-lts",
Description: evetest.TestParameterDescription{
Summary: "EVE version to upgrade from (the pre-upgrade initial version)",
Default: "16.0.0-lts",
},
}
evetest.TestParameterDefinition{
Key: initialHypervisorParamKey,
DefaultValue: evetest.HypervisorKVM,
Description: evetest.TestParameterDescription{
Summary: "Hypervisor used by the initial (pre-upgrade) EVE version",
Default: "kvm",
AllowedValues: "kvm|xen|kubevirt",
},
}
And reuse among tests.
milan-zededa
reviewed
Aug 4, 2026
| evetest.HypervisorParameter(), | ||
| evetest.TPMParameter(), | ||
| evetest.DiskSizeMiBParameter(), | ||
| evetest.TestParameterDefinition{ |
Contributor
There was a problem hiding this comment.
I think the device RAM size and CPU count parameters are something we'll want to reuse in other tests as well. I'd suggest moving these parameter definitions into testparam.go (like we have evetest.TPMParameter(), evetest.DiskSizeMiBParameter(), etc.).
milan-zededa
reviewed
Aug 4, 2026
| }) | ||
| device.ApplyConfig(devConfig, false, false) | ||
|
|
||
| // The conversion is several reboots plus an EVE-k bring-up and runs close to |
Contributor
There was a problem hiding this comment.
Could you please also check TestEVEUpgrade -- do we need to change/add anything in the test when the target is eve-k?
Or should we forbid kvm->k combination in TestEVEUpgrade and use it only for testing upgrades within the same hypervisor?
milan-zededa
reviewed
Aug 4, 2026
| } | ||
| initialHypervisor := evetest.GetTestParameter[evetest.Hypervisor](initialHypervisorParamKey) | ||
| convVersion := evetest.GetEVEVersionParameterValue() | ||
| targetHypervisor := evetest.GetHypervisorParameterValue() |
Contributor
There was a problem hiding this comment.
Do we want the initial and target hypervisors to be configurable? I mean isn't this test specifically for kvm->k?
milan-zededa
reviewed
Aug 4, 2026
| // volverify does not have to be shipped in: containerd has already unpacked the app | ||
| // image, so the same statically-linked binary the app would run is on disk. Args: | ||
| // seed, ops, max-blocks, expect-committed. | ||
| const devsideVerifyScript = `set -u |
Contributor
There was a problem hiding this comment.
Would it make sense to ship this script inside the volverify app? (I mean to put it to testapps/volverify/scripts)
Contributor
There was a problem hiding this comment.
(jusk asking, I see that there are many more scripts defined inside the test)
Contributor
There was a problem hiding this comment.
It would definitely make sense to make them part of the app if we might ever need to troubleshoot the upgrade and run them manually inside the app.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two evetest integration tests for the in-field EVE-kvm→EVE-k boot-disk conversion,
plus the test app one of them needs.
TestKvmToKResizedrives the conversion end to end: a small-geometry EVE-kvm base(e.g. 16.6.0) → the current EVE-kvm build → the current EVE-k build, where the
cross-flavor jump forces the offline shrink+grow. The boot disk is sized with no
free tail so the conversion has to shrink
/persistrather than grow into slack,and the test asserts the decision really was a shrink. Unlike the eden escript it
mirrors, which deletes the app before the k update, this keeps a container app and
its data volume across the conversion, so the run also covers the volume
relocation.
TestAppVolumeShrinkCorruptionasks what the shrink does to an app's data volume.It fills
/persistso the volume lands in the region the shrink must evacuate,writes a verifiable pattern into it, runs the conversion — optionally letting the
watchdog interrupt it — and then judges the volume with fsck plus a replay of the
pattern. The verdict is taken on the volume itself and does not depend on the app
coming back, because the post-conversion app wedge has repeatedly cost otherwise
complete runs their result.
evetest/testapps/volverifyis the app that writes and replays that pattern: adeterministic seeded block stream with an on-volume committed index, so an
interrupted write is distinguishable from actual corruption. It carries its own Go
unit tests and a loopback-ext4 script for iterating on it off-device.
Both tests provision from the live image rather than the installer. The
installer ESP carries a 0-byte
boot/.boot_repository, which the offline grow'sFAT32 copy rejects with
invalid start cluster: 0(diskfs/go-diskfs#417). That isnow fixed upstream in diskfs/go-diskfs#419, but
pkg/storage-resizeris not yetre-pinned onto a go-diskfs containing the fix, so the live-image path is still
required.
Draft: the branch is not squashed and is behind master; the parameter sweeps and
soak plumbing are still moving.
PR dependencies
Depends on #6063 — the conversion itself (
pkg/storage-resizerand the baseosmgrside) is not in master, so these tests can only run against a build that carries
it.
How to test and validate this PR
Test-framework only; nothing here ships on a device. Build an EVE image that
includes #6063, then:
TestKvmToKResizeneedsINITIAL_EVE_VERSIONpointing at a small-geometry base(e.g.
16.6.0) andDISK_SIZE_MBsmall enough that there is no free tail —otherwise the conversion grows instead of shrinking and the test fails its
decision==shrinkassertion rather than silently passing. It also enforcesresource floors (16 GiB RAM, 8 vCPUs); EVE-k plus Longhorn does not converge on
the framework defaults.
TestAppVolumeShrinkCorruptiontakesDATAVOL_MB,FILL_PEAK_PCT,FILL_KEEP_GIBandVOLVERIFY_SEED/VOLVERIFY_OPS. A failing volume verdict isthe finding being hunted, so a run that finds nothing to verify is reported as
such rather than as a pass.
volverify itself can be exercised without a device:
go test ./evetest/testapps/volverify/... ./evetest/testapps/volverify/scripts/loopback-ext4-test.shChangelog notes
No user-facing changes. Test-framework only (
evetest/); nothing underpkg/orthe EVE rootfs is touched.
PR Backports
Checklist
I've provided a proper description
I've added the proper documentation
I've tested my PR on amd64 device
I've tested my PR on arm64 device
I've written the test verification instructions
I've set the proper labels to this PR
I've checked the boxes above, or I've provided a good reason why I didn't
check them.
Draft PR, still under validation: the runs behind it are amd64 QEMU only, and the
device-testing boxes stay unchecked until the sweeps settle. Nothing here runs on
a device outside the test framework.