diff --git a/AGENTS.md b/AGENTS.md index e30afb270b..b9fb6a9ba5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -198,6 +198,51 @@ All hooks managed by [prek](https://prek.j178.dev/) (installed via `npm install` 6. Use `./scripts/dev-setup.sh --expose-cli` only with explicit approval for host-visible CLI exposure 7. Run the tests targeted to the behavior you change once per relevant change set; rerun them after later edits or hook autofixes that can affect that behavior +### E2E Selection and Authoring + +Use live E2E only for behavior that needs a real shell, installer, process, +Docker, OpenShell, `/proc`, sandbox, external service, or GitHub Actions +boundary. Put deterministic code, parser, registry, workflow-planner, and +fixture logic in unit, integration, package-contract, or `e2e-support` tests +instead. Do not add a live E2E target for a check that can be observed through a +stable local boundary. + +Before adding or extending E2E coverage, name the semantic coverage dimension +that is missing. Existing migrated examples show the intended granularity: +catalogue targets pair environment, onboarding profile, expected state, optional +lifecycle, and `suiteIds`; `dashboard-remote-bind` owns install, onboard, +artifacts, and terminal cleanup; `credential-sanitization`, +`telegram-injection`, `messaging-providers`, `messaging-compatible-endpoint`, +and `gpu-e2e` are separate behavior contracts rather than one broad "full" run. +Extend matrix metadata only when it selects an already-defined behavior +dimension. Do not duplicate behavior logic in a second registry, workflow list, +or hand-maintained catalogue; use the typed registry and shared E2E workflow +planner documented in [`test/e2e/README.md`](test/e2e/README.md) and +[`test/e2e/docs/README.md`](test/e2e/docs/README.md). + +If a gap is real but not ready for a test, record it as a combinatorial gap +instead of adding speculative coverage. State the missing dimension, the +existing nearest coverage, why a new test would duplicate or overreach current +behavior, and the issue or PR that will make it testable. A gap note must not +change release judgment by itself. + +Assert outcomes, state, artifacts, and redacted diagnostics. Do not assert +incidental terminal output, progress wording, spinner frames, ANSI escape +sequences, timing text, or prompt layout unless that text is the product +contract under review. Terminal traces are evidence; they are not stable +behavior unless the issue explicitly makes them the behavior. + +Retries require a checked-in bounded policy with a narrow transient signature, +owner, idempotence or reconciliation basis, attempt evidence, and an entry in +[`test/e2e/RETRY_INVENTORY.md`](test/e2e/RETRY_INVENTORY.md). Do not add +unproven retries, ambiguous mutation retries, or broad failed-job reruns. A +mutation retry is allowed only after the test reconciles the external state and +proves repeating the same desired operation is safe. Keep bounded operation +retries separate from complete workflow reruns: `E2E / Main Retry` records +attempts and does not request a broad rerun, while Hosted Runner Recovery owns +at most one full rerun only for authenticated GitHub-hosted runner-loss +evidence. + ### Plain Language and Direct Design - Use existing repository vocabulary and name what a thing does. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 695169a030..c0295e6eb1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -262,6 +262,34 @@ artifact sink. Pass the auto fixture's frozen, canonical `progress` capability through unchanged; custom, copied, or no-op progress adapters are rejected at audited subprocess boundaries. +Use live E2E only when the behavior needs a real shell, installer, process, +Docker, OpenShell, `/proc`, sandbox, external service, or GitHub Actions +boundary. Prefer unit, integration, package-contract, or `e2e-support` tests for +deterministic code and workflow-planner logic. Select or extend coverage by +semantic dimension, not by incidental output. Migrated targets such as +`dashboard-remote-bind`, `credential-sanitization`, `telegram-injection`, +`messaging-providers`, `messaging-compatible-endpoint`, and `gpu-e2e` show the +expected shape: each target owns a behavior contract, while the typed registry +keeps environment, onboarding profile, expected state, lifecycle, and `suiteIds` +as matrix metadata. Extend that metadata only to select existing behavior; do +not duplicate behavior logic in workflows, lists, or catalogues. When a +combination is missing but not ready for a test, record a combinatorial gap with +the missing dimension, nearest existing coverage, why a new test would duplicate +or overreach current behavior, the follow-up owner, and the issue or PR that will +make the gap testable. Do not add speculative coverage or change release judgment. + +E2E assertions should check outcomes, state, artifacts, and redacted diagnostics. +Do not assert incidental terminal output, progress wording, ANSI escape +sequences, spinner frames, or timing text unless that text is the product +contract. A retry must have a checked-in bounded policy, a narrow transient +signature, idempotence or reconciliation evidence, per-attempt artifacts, and a +matching [`test/e2e/RETRY_INVENTORY.md`](test/e2e/RETRY_INVENTORY.md) entry. Do +not add unproven retries, ambiguous mutation retries, or broad failed-job +reruns. Keep operation-level retries separate from complete workflow reruns: +`E2E / Main Retry` records attempt evidence without requesting a broad rerun, +and Hosted Runner Recovery owns at most one full rerun only for authenticated +GitHub-hosted runner-loss evidence. + ### macOS Test Dependencies Some tests run command-line tools that macOS does not ship.