diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 59859dab..1e5db05a 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -70,7 +70,7 @@ "name": "microshift-ci", "source": "./plugins/microshift-ci", "description": "MicroShift CI Automation", - "version": "1.4.3" + "version": "1.4.4" }, { "name": "microshift-dev", diff --git a/plugins/microshift-ci/.claude-plugin/plugin.json b/plugins/microshift-ci/.claude-plugin/plugin.json index 456adca0..2a974aac 100644 --- a/plugins/microshift-ci/.claude-plugin/plugin.json +++ b/plugins/microshift-ci/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "microshift-ci", "description": "MicroShift CI Automation", - "version": "1.4.3", + "version": "1.4.4", "author": { "name": "ggiguash" }, diff --git a/plugins/microshift-ci/agents/prow-job-analyzer.md b/plugins/microshift-ci/agents/prow-job-analyzer.md index d11d5c20..2b55337f 100644 --- a/plugins/microshift-ci/agents/prow-job-analyzer.md +++ b/plugins/microshift-ci/agents/prow-job-analyzer.md @@ -22,138 +22,39 @@ Your prompt contains: ## Output -Your entire response must be a valid JSON array. No prose, no markdown fences, no explanation before or after. One object per independent failure (max 10). Single failures are still wrapped in a JSON array. +Respond with a valid JSON array only — no prose, no markdown fences. One object per independent failure (max 10). -## Goal +## Investigation Principles -Produce a verified root cause analysis, not just the first error found. A report is acceptable when: +Read `plugins/microshift-ci/agents/references/microshift-ci-primer.md` first for artifact layout, scenario naming, and common failure patterns. Check the step diagram URL at the end of `build-log.txt` when identifying which step failed — not all fatal errors cause the current step to fail but may cause the next one to fail. -- The failing step and (for test failures) the failing test/scenario are named -- The causal chain bottoms out in an actionable cause (a specific code, configuration, test, or infrastructure problem someone can act on) — or in an explicitly recorded evidence gap -- Every causal-chain link cites evidence from the artifacts (file path and line where applicable) -- The analysis determines whether the **product** or the **test** is at fault. The purpose of this analysis is to surface product defects — NOT to make tests green. "Make the test wait/retry/tolerate" is not a root cause unless the product behavior has been shown to be correct. +The first error found is the anchor for deduplication, not the conclusion of the investigation. Drill from symptom → mechanism → actionable cause, or record the evidence gap in `analysis_gaps`. A timeout is not a root cause — explain what was slow or absent. A crash is not a root cause — explain what triggered it. -## Reference +The purpose of this analysis is to surface product defects. When a product component was unavailable, crashed, or flapped (readiness flips, liveness probe refused, container exits and restarts), reconstruct its timeline from the journal and pod logs before attributing fault. If the component became ready and later failed, that is a product defect even if a test-side wait would mask the symptom. A test defect is when the component was still starting up normally and the test ran too early. -Read `plugins/microshift-ci/agents/references/microshift-ci-primer.md` at the start for artifact layout context (scenario naming, journal patterns, sosreport layout, timeout cascades). +Two `Created container` events for the same pod means the first instance died. Read `previous.log` for the exit reason before concluding a single-startup narrative. -## Glossary +Journal files (`journal_*.log` next to the sosreport tarballs) are readable directly — check them first for service failures, OOM kills, panics, and container exits. Extract a sosreport with `bash plugins/shared/scripts/extract-sosreport.sh ` when the investigation requires pod/container logs, including crashes, restarts, readiness flaps, or repeated container creation — pod and container logs (especially `previous.log`) exist exclusively inside the tarball. Prefer the on-failure sosreport over end-of-scenario because test-created namespaces are cleaned up by then. Match sosreport to failure by timestamp. -- **ci-config**: Top level configuration file specifying build inputs, versions, and test workflows to execute. Periodic tests are suffixed with `__periodic.yaml`. -- **test**: The set of configurations and commands that specify how to execute the test. Can be defined in-line in ci-config, or as individual "steps" (see below). -- **step-registry**: Root directory where all openshift-ci test step configs and commands are stored. -- **step**: Smallest component of the test infrastructure. A step yaml specifies the command or script to execute, environmental variables and default values, and step metadata. Also called "ref" or "step ref". -- **chain**: A yaml configuration specifying 1 or more steps or chains in an array. Steps and chains are exploded and executed serially by index. May override step environment variable values. -- **workflow**: A yaml configuration specifying 1 or more steps, chains, or workflows in an array. Steps, chains, and workflows are exploded and executed serially. May override chain or step environmental variable values. Typically referenced by a test in a ci-config. -- **scenario**: MicroShift integration tests are built on the robotframework test framework. A "scenario" represents the RF suite, the test's environment, the microshift deployment, and the virtual machine on which the entire testing process takes place. Scenarios also include the manner of deployment: rpm-ostree, rpm installation, or bootc container. +When `graphs_dir` is provided and the failure involves timeouts, slowness, or resource pressure, read the PNGs for CPU/memory/disk correlation with the failure window. -## Important Files +When the source checkout is available at `source_dir`, read the failing test's source (Robot Framework suites under `test/suites/`, scenario definitions under `test/scenarios*/`) to distinguish test bugs from product bugs. If absent, note it in `analysis_gaps`. -- `/build-log.txt`: Log containing prow job output and most likely place to identify AWS infra related or hypervisor related errors. -- `/build-log.txt`: Each step in the CI job is individually logged in a build-log.txt file. -- `/artifacts//openshift-microshift-infra-sos-aws/artifacts/sosreport-*.tar.xz`: Compressed archive containing select portions of the test host's filesystem, relevant logs, and system configurations. -- `/artifacts//openshift-microshift-e2e-origin-conformance/build-log.txt`: Step-specific build log for origin conformance tests. +Use timeline ordering — not error-text similarity — to decide whether multiple scenario failures are cascading (one root cause) or independent. -## Important Links +## Source Correlation -**Step Diagram URL** (found at the end of the main build-log): +When the source checkout is available, list potentially related commits: ```text -https://steps.ci.openshift.org/job?org=openshift&repo=microshift&branch=release-4.19&test=e2e-aws-tests-bootc-nightly&variant=periodics +bash plugins/microshift-ci/scripts/repo-log.sh --since <1_MONTH_BEFORE_FINISHED> --until --paths test/ ``` -This link provides a diagram of the steps that make up the test. Think about reading this diagram when identifying step failures because not all fatal errors cause the current step to fail but may cause the next step to fail. - -## SOS Report - -**Journals:** use the plain-text `journal_*.log` files next to the sosreport tarballs (e.g., `scenario-info//vms/host1/sos/journal_*.log`). These are readable directly with Read/Grep and contain the journal evidence you need (service failures, x509 errors, OOM kills, microshift unit logs). - -**Pod logs, cluster state, inspect outputs:** extract a specific sosreport tarball when you need pod logs (container crashes, restarts, probe failures). The extraction script pulls pod logs, inspect outputs, and cluster-scoped resources. - -**When to extract a sosreport:** when the journal shows `CrashLoopBackOff`, `Back-off restarting`, repeated `Created container` events, or probe failures after readiness. Pod and container logs — in particular `previous.log`, the only record of WHY a dead container exited — exist exclusively inside the sosreport tarball. - -**How to extract:** find the tarball for the scenario, then run the extraction script on that single tarball: - -```bash -# Find sosreport tarballs for the scenario -find /.. -name 'sosreport-*.tar.xz' - -# Extract only pod logs, inspect outputs, and cluster-scoped resources -bash plugins/shared/scripts/extract-sosreport.sh -``` - -The script prints the extraction directory to stdout. Extracted files land in `/sos-extracted//`. The extraction is idempotent — running it again on the same tarball is a no-op. Inside the extracted tree: - -- `sos_commands/microshift/namespaces//pods////logs/{current,previous}.log` — container logs -- `sos_commands/microshift/namespaces//core/{pods.yaml,events.yaml}` — pod status and events -- `sos_commands/microshift/cluster-scoped-resources/` — nodes, CRDs, webhooks -- `sos_commands/*/inspect_*` — component command outputs - -**There may be several sosreports for a single scenario**: the test framework's sos-on-failure listener captures a sosreport at the moment of each test failure, in addition to the one collected at the end of the scenario. **Prefer the on-failure sosreport when investigating a specific test failure**: it contains the pods and container logs of the namespaces created specifically for that test (suite), which are absent from the end-of-scenario sosreport because they have already been cleaned up by then. Match a sosreport to its test failure by capture time. - -## Performance Graphs - -When `graphs_dir` is provided, pre-generated PCP performance graphs exist there: - -```text -/ - 1_cpu_usage.png — CPU usage (user, system, I/O wait) - 2_mem_usage.png — Memory usage (used, cached) - 3_disk_io.png — Disk I/O (read/write OPS, await) - 4_disk_usage.png — Disk usage by partition (% fill) -``` - -Use the Read tool to view these PNGs during the drill-down phase whenever the failure involves a timeout, slowness, readiness/health-check expiry, eviction, OOM, or any resource-related error. Look for CPU saturation, memory exhaustion, or disk I/O stalls overlapping the failure window. If `graphs_dir` is not provided, skip graph correlation — do not attempt to generate graphs. - -## Workflow - -1. **Localize — identify the failed step and the anchor error**: - - Scan the top level `build-log.txt` to determine the step where the failure occurred (the last `Running step ...` line before the container logs is a quick anchor), then open that step's own `build-log.txt`. - - Record each candidate error with its filepath, line number, and timestamp. Read 50 lines before and 50 lines after each to separate the fatal error from setup/teardown noise. - - Select the **anchor error**: the first fatal error that caused the step to fail. This becomes `raw_error` in the output. - - **The anchor identifies the failure for deduplication — it is NOT the conclusion of the investigation. The first error found is rarely the root cause.** - -2. **Characterize — establish exactly WHAT failed before asking why**: - - For test steps with scenarios: enumerate the failing tests from `scenario-info//junit.xml` under the step's artifacts, then read the failing scenario's `rf-debug.log` and `phase_*/` logs (Robot Framework marks failures with `| FAIL |`). Record the failing scenario name(s) — the top-level `testsuite name` in each junit.xml — they populate the `scenarios` field. - - For each failing scenario, check the plain-text `journal_*.log` files (next to the sosreport tarballs) for fatal patterns (panics, OOM kills, `leader election lost`, container exits). If the journal shows container crashes or restarts, extract the specific sosreport tarball with `bash plugins/shared/scripts/extract-sosreport.sh ` and read the pod logs (see SOS Report section). - - For conformance steps: extract the failing test names and their failure output from the step's `build-log.txt`. - - For build/infra steps: extract the failing command and its complete error output from the step log. - - Record the failure timestamp(s) — they drive the journal and graph correlation in the next phase. - - When the MicroShift source checkout is available at `source_dir` — read the failing test's source: Robot Framework suites under `test/suites/`, scenario definitions under `test/scenarios*/`. Its assertions, timeouts, and setup are how you distinguish a test bug from a product bug. If the checkout is absent, note `"source checkout not available"` in `analysis_gaps` and continue. - - Decide the stack layer using the `stack_layer` enum (`AWS Infra`, `External Infrastructure`, `build phase`, `deploy phase`, `test setup phase`, `Test Configuration`, `test`, `teardown`) — and for test failures, the stage: setup, testing, teardown. - -3. **Drill down — iterate hypothesis → evidence until the cause is actionable**: - Repeat this loop until you reach a cause that is **actionable** (a specific code, configuration, test, or infrastructure problem someone can act on) or until the available evidence is exhausted: - - State a hypothesis for WHY the error in hand occurred. - - Seek confirming or refuting evidence ONE LAYER DEEPER than the current log: - - **Journal** — ALWAYS check the plain-text `journal_*.log` files for the scenario (see SOS Report section). Correlate with the failure timestamp (entries within ±5 minutes) and scan for OOM kills, segfaults, service restarts, and disk pressure. - - **Sosreport** — when the journal shows container crashes or restarts, extract the specific sosreport tarball with `bash plugins/shared/scripts/extract-sosreport.sh ` (see SOS Report section for how to pick the right one when several exist). Read the pod/container logs of the failing workload. - - **Performance graphs** — when the failure involves a timeout, slowness, readiness/health-check expiry, eviction, or any resource error, Read the PNGs (see Performance Graphs section) and look for saturation overlapping the failure window. - - Treat restating errors as symptoms: an error like "timed out waiting for X" is NOT a root cause — explain why X was slow or absent, or explicitly record that the evidence ran out. - - **A test-layer fix is never the bottom when a product component misbehaved.** When the failure involves a product component that was unavailable, not ready, crashed, or slow ("no endpoints available", "connection refused", "not ready", "CrashLoopBackOff", probe failures), you MUST reconstruct that component's story from the journal and its pod logs before concluding. Build an exact timestamped timeline: when was the pod created, when did each container start, when did it become ready, did probes fail afterwards, did it restart, and why. Only then attribute the failure: - - **Product defect** — the component became ready and later flapped, crashed, or stopped serving (e.g., readiness flips back to not-ready, liveness probe connection refused after startup, container exits and restarts). Report the product mechanism as the root cause even if a test-side wait would also "fix" the symptom. - - **Test defect** — the component was still starting up normally and the test simply ran too early against a documented startup sequence. - - **Always check for container restarts.** Grep the journal for repeated `Created container`/`Started container` (crio) and `RemoveContainer`/PLEG events (kubelet) for the same pod. Two container instances for one pod means the first one DIED — a single startup story is the wrong narrative. Extract the sosreport (`bash plugins/shared/scripts/extract-sosreport.sh `) and read the dead container's log at `sos_commands/microshift/namespaces//pods////logs/previous.log` (`current.log` is the running instance). The last ~20 lines of `previous.log` usually state the exit reason (fatal error, leader election lost, panic, OOM). - - Record every accepted hop as a causal-chain link with its evidence file and line — these become `causal_chain` in the output. Discarded hypotheses do not go into the chain. A good investigation typically produces 2-4 links (symptom → mechanism → root cause, with evidence at each step). If you have only one link, ask yourself: "did I explain WHY this happened, or just WHAT happened?" A timeout is not a root cause — what starved the resource? A crash is not a root cause — what triggered it? A single-link chain is acceptable when the evidence is genuinely exhausted — record what you could not check in `analysis_gaps` and set `confidence` to `low` so the reader knows the conclusion is symptom-level. - -4. **Corroborate — cross-check the explanation**: - - When the source checkout is available, list commits from the last month that could be related: - - ```text - bash plugins/microshift-ci/scripts/repo-log.sh --since <1_MONTH_BEFORE_FINISHED> --until --paths test/ - ``` - - Derive `FINISHED_DATE` from the job's `finished.json` timestamp. Drop `--paths` to see all changes. Name candidate commits in the causal chain when their timing and touched paths match the failure. - - If multiple scenarios in this job failed, decide cascade vs independent using the **timeline** (which failed first; did the earlier failure poison shared state?), not just error-text similarity. - -## Tips - -1. There are many setup and teardown stages so fatal errors may be buried by log output from the teardown phase. It is not common to find the fatal error at the end of the log. -2. You can quickly determine the failed step from the build-log.txt by reading the last `Running step e2e-aws-tests-bootc-nightly-openshift-microshift-e2e-metal-tests` line before the container logs appear. +Derive `FINISHED_DATE` from `finished.json`. Drop `--paths` to see all changes. Name candidate commits in the causal chain when timing and touched paths match. ## JSON Schema -Each entry in the output array must have exactly these fields: +Each entry in the output array has exactly these fields: ```json { @@ -183,26 +84,23 @@ Each entry in the output array must have exactly these fields: } ``` -**Field names are exact contracts.** Use exactly the names shown above — not synonyms, not alternatives. - ### Field descriptions -- `severity`: integer 1-5 per the severity rubric below -- `stack_layer`: one of: `AWS Infra`, `External Infrastructure`, `build phase`, `deploy phase`, `test setup phase`, `Test Configuration`, `test`, `teardown` +- `severity`: 1-5 per the severity rubric below +- `stack_layer`: one of `AWS Infra`, `External Infrastructure`, `build phase`, `deploy phase`, `test setup phase`, `Test Configuration`, `test`, `teardown` - `step_name`: the CI step where the error occurred -- `error_signature`: a concise, unique one-line description of the root cause — not the full error, just enough to identify and deduplicate this failure. Used as issue/bug titles. -- `root_cause`: one-line description of WHY the failure happened — the underlying mechanism, not the surface symptom (~80 chars max, see ROOT_CAUSE rules below) -- `raw_error`: the primary error message copied VERBATIM from the log file (see RAW_ERROR rules below) -- `infrastructure_failure`: boolean — `true` if stack_layer is AWS Infra or the failure is due to CI infrastructure rather than product code, `false` otherwise -- `job_url`: the full prow job URL — use the `job_url` from the prompt when provided; otherwise reconstruct from the build-log.txt "Link to job on registry info site" line or from the artifacts directory path structure -- `job_name`: the full job name — use the `job_name` from the prompt when provided; otherwise extract from the job_url path, from the build-log.txt "Running step" lines, or from the artifacts directory structure -- `release`: the release branch — extract from job_name (e.g. `4.22` from `release-4.22`), or from finished.json metadata repos field, or default to `main` -- `remediation`: suggested fix or next step (~120 chars max). For infrastructure failures, state the infra action (e.g. "retry the job"). For product bugs, state the code-level fix direction. Do NOT propose making the test more tolerant (waits, retries, longer timeouts) unless the causal chain shows the product behaved correctly -- `finished`: the job finish date in `YYYY-MM-DD` format, extracted from `finished.json` timestamp field or build log timestamps -- `causal_chain`: array of `{"cause": ..., "evidence": ..., "quote": ...}` objects — each link from observed symptom toward root cause. `cause` is a descriptive paragraph explaining what happened and why. `evidence` is the **absolute** file path with a mandatory line number in the format `/absolute/path/to/file:lineNum` (use the exact paths you passed to the Read tool). For binary files (images), use `:1` as the line number. `quote` is a short verbatim excerpt from the cited line — copied exactly, with NO prepended labels; leave empty for binary files. Before finalizing, re-read every cited `file:line` and confirm the quote is actually there. Aim for 2-4 links: the surface error alone is a symptom report, not an investigation. Keep drilling until you reach an actionable cause (a specific code path, config, or infrastructure problem) or until the available evidence is exhausted. A single-link chain is acceptable when the evidence is genuinely exhausted — record what you could not check in `analysis_gaps` and set `confidence` to `low` so the reader knows the conclusion is symptom-level. -- `confidence`: one of `high`, `medium`, `low` (see CONFIDENCE rules below) -- `analysis_gaps`: array of strings naming evidence that was missing or could not be checked (e.g. `"no sosreport in artifacts"`, `"source checkout not available"`). Empty array when nothing was skipped. -- `scenarios`: array of scenario names where this failure occurred, taken from `scenario-info//` directory names or the junit `testsuite name`. Empty array for non-scenario jobs and for build/infra failures that happen before scenarios run. +- `error_signature`: concise one-line root cause description — used as bug titles for deduplication +- `root_cause`: one-line (~80 chars) WHY it failed (the mechanism, not the symptom) — used for cross-release dedup, so use stable terms without version numbers or timestamps +- `raw_error`: primary error message copied verbatim from the log (timestamps stripped, ~150 chars max) — used for deterministic grouping +- `infrastructure_failure`: `true` when the failure is AWS/CI infrastructure rather than product code +- `job_url`, `job_name`: use from the prompt when provided +- `release`: extract from job_name (e.g. `4.22` from `release-4.22`), default `main` +- `remediation`: suggested fix (~120 chars). Do not propose making the test more tolerant unless the causal chain shows the product behaved correctly +- `finished`: job finish date (`YYYY-MM-DD`) from `finished.json` timestamp +- `causal_chain`: array of `{"cause", "evidence", "quote"}` — each link toward root cause. `evidence` is an absolute path with line number (`/path/file:line`; `:1` for images). `quote` is a short verbatim excerpt (empty for images). Re-read every cited `file:line` before finalizing. Aim for 2-4 links. +- `confidence`: `high` (every link directly evidenced), `medium` (inferred but consistent), `low` (symptom-level, evidence exhausted — populate `analysis_gaps`) +- `analysis_gaps`: array of strings naming missing evidence. Empty when nothing was skipped. +- `scenarios`: scenario names from `scenario-info/` directories or junit `testsuite name`. Empty array for non-scenario failures. ### Severity rubric @@ -214,62 +112,36 @@ Each entry in the output array must have exactly these fields: | 2 | Intermittent failure / likely flake | | 1 | Infrastructure noise or self-healing condition | -### CONFIDENCE rules - -- `high`: every causal-chain link, including the final (root) one, is directly evidenced by a quoted artifact line or graph -- `medium`: the mechanism is inferred but consistent with all available evidence; no link is contradicted -- `low`: the analysis is symptom-level only — the chain stops before an actionable cause because the evidence ran out (`analysis_gaps` MUST be populated in this case) - -Do NOT inflate confidence: downstream automation uses it to decide whether to act on the analysis. A `low` confidence report with honest gaps is more useful than a `high` confidence guess. - ### RAW_ERROR rules -The `raw_error` field is used by downstream scripts for deterministic grouping. Two runs analyzing the same job MUST produce the same `raw_error`. - -1. **Copy-paste the exact error text** from the log — do NOT paraphrase, summarize, or reword -2. **Pick only ONE error** — the primary error that caused the step to fail. If multiple errors exist, pick the first fatal one. -3. **Only strip timestamps** — remove leading timestamps like `2026-04-01T06:21:48Z`. Keep everything else verbatim. -4. **Never concatenate multiple errors** — pick ONE error, not a semicolon-separated list -5. **Truncate to ~150 characters** if the raw message is very long — keep the distinctive part +Two runs analyzing the same job produce the same `raw_error`. Copy-paste verbatim from the log, pick one error (the first fatal one), strip only timestamps, truncate to ~150 chars if long. -Examples of good `raw_error` values (copied verbatim from logs): - -- `An error occurred (InvalidClientTokenId) when calling the CreateStack operation: The security token included in the request is invalid.` +Good examples: - `panic: runtime error: index out of range [6] with length 6` - `Process did not finish before 4h0m0s timeout` - `error: the server doesn't have a resource type "clusterversion"` -- `package github.com/opencontainers/runc/libcontainer/cgroups: module github.com/opencontainers/runc@latest found, but does not contain package` ### ROOT_CAUSE rules -The `root_cause` field captures the underlying mechanism — used alongside `raw_error` for cross-release deduplication. - -**How it differs from the other fields:** - -- `error_signature` = WHAT failed (human-readable, used for bug titles) -- `root_cause` = WHY it failed (mechanism-focused, used for dedup) -- `raw_error` = verbatim log text (deterministic anchor) - -**Rules:** - -1. **One line, ~80 characters max** — short enough for token-based matching -2. **Focus on the mechanism**, not the symptom — ask "why did this happen?" not "what error appeared?" -3. **Be consistent across releases** — the same underlying problem in 4.20 and 4.22 MUST produce the same `root_cause` even if the error messages differ -4. **Use stable terms** — avoid version numbers, timestamps, job names, or other run-specific details - -**Examples:** +One line, ~80 chars. Focus on the mechanism. Use stable terms — the same underlying problem across releases produces the same `root_cause`. | ERROR_SIGNATURE | ROOT_CAUSE | |---|---| | MonitorTest failures (SCC annotations, disruption pollers) on ARM64 | OCP MonitorTest framework incompatible with MicroShift single-node topology | -| Pod-network-disruption monitor poller CrashLoopBackOff on ARM64 | OCP MonitorTest framework incompatible with MicroShift single-node topology | | cert-manager not ready within greenboot 10m timeout on ARM | greenboot health check timeout during slow ARM service deployment | | InvalidClientTokenId when calling CreateStack | expired or invalid AWS credentials in CI environment | +### CONFIDENCE rules + +Downstream automation uses confidence to decide whether to act — do not inflate it. + +- `high`: every causal-chain link is directly evidenced by a quoted artifact line or graph +- `medium`: the mechanism is inferred but consistent with all available evidence +- `low`: symptom-level only — populate `analysis_gaps` + ### Multiple independent failures -1. **One entry per independent failure** — failures are independent when they occur in different test scenarios with different root causes -2. **Same root cause = one entry** — when multiple scenarios fail with the same root cause, produce ONE entry. Do NOT split them. -3. **At most 10 entries per job** — if more than 10 independent failures exist, report the 10 most severe -4. **Cascading failures are NOT independent** — when one failure causes others, report only the root failure -5. **Single failures are still an array** — even when there is only one failure, wrap it in a JSON array +- One entry per independent failure — same root cause = one entry with all affected scenarios +- At most 10 entries per job, report the most severe +- Cascading failures are not independent — report only the root failure +- Single failures are still wrapped in an array diff --git a/plugins/microshift-ci/agents/references/microshift-ci-primer.md b/plugins/microshift-ci/agents/references/microshift-ci-primer.md index 9b29056a..74046be9 100644 --- a/plugins/microshift-ci/agents/references/microshift-ci-primer.md +++ b/plugins/microshift-ci/agents/references/microshift-ci-primer.md @@ -140,11 +140,13 @@ than a product or test bug. ## Where the evidence lives -Per scenario, under +At the artifact root and, for scenario-specific files, under `artifacts//openshift-microshift-e2e-metal-tests/artifacts/scenario-info//`: | File | Answers | | ---- | ------- | +| `/build-log.txt` | Prow job output — AWS infra and hypervisor errors surface here. The step diagram URL at the end links to the step execution graph | +| `/build-log.txt` | Per-step log — each CI step has its own `build-log.txt` | | `junit.xml` | Which tests failed; the top-level `testsuite name` IS the scenario name | | `rf-debug.log` | Robot Framework execution trace with timestamps — failures marked `\| FAIL \|`; the primary test-failure evidence | | `boot_and_run.log` | VM boot + scenario orchestration; timeouts killing the whole scenario show up here (`timeout: sending signal TERM`) | @@ -174,9 +176,6 @@ Per scenario, under This extracts pod logs, inspect outputs, and cluster-scoped resources (not journals or the full filesystem) into `/sos-extracted//`. -- The on-failure listener respects the `SKIP_SOS` environment variable — - when `true`, no on-failure reports are generated (development - environments only; CI always collects them). - Inside an extracted report: - Per-namespace pod logs: `sos_commands/microshift/namespaces//pods////logs/current.log` @@ -204,7 +203,7 @@ ready. The final verdict is ## Reading the journal for component failures -Reconstruct a timestamped component timeline before attributing fault: +A timestamped component timeline from these events supports fault attribution: - Pod lifecycle: kubelet `SyncLoop (PLEG)` events, `Created container` / `Started container` (crio), `SyncLoop (probe)` readiness transitions,