New --all-reruns-need-to-pass argument#311
Conversation
This addresses your requirement to verify that non-deterministic tests (that work ~90% of the time) pass consistently by requiring all reruns to pass after an initial failure. Signed-off-by: Manuel Saelices <msaelices@gmail.com>
|
Thank you for your PR. Unfortunately I do not understand which problem it solves could you please explain your use case with an example? Additionally the merge conflict and lint errors could be solved. |
|
@msaelices Are you still interested in this PR, otherwise I'll close it as abandoned. |
We are using it to run evals that are not deterministic, like LLM based Scenario: Let's say you have 100 evals with, e.g. 5% of failure each one of them, which is ok for you. Even with a seemingly "good" 5% per-eval failure rate, running 100 evals makes it very likely, with a ~99.4% chance, that at least one fails. How do you differentiate between a harmless flaky failure and an actual regression, where the failure rate jumps from something acceptable like 5% to, for example, 50%? We use For example, with |
…o-pass-argument # Conflicts: # src/pytest_rerunfailures.py # tests/test_pytest_rerunfailures.py
|
@msaelices Did I understand correctly? When using |
Exactly. It enforces that all follow-up reruns must pass for the test to be considered successful. |
A plugin can now return a RerunPolicy for a test's first failure to give that specific failure class its own rerun count / all-reruns-need-to-pass mode and an outcome tag, without changing how other failures are rerun. Motivation: distinguish transient provider-infra failures (5xx / 429 / timeouts) from behavioral failures in an eval suite — give infra failures a single rerun with pass-on-recovery semantics, and tag the ones that persist so a sustained outage can be reported as "infra-degraded" instead of a wall of red — while behavioral failures keep the strict --reruns N --all-reruns-need-to-pass net. This per-failure-type policy can't be expressed via --only-rerun (it is bypassed under --all-reruns-need-to-pass) or a global --reruns count. - New hookspec pytest_rerunfailures_rerun_policy(item, report, call) in a dedicated pytest_rerunfailures_newhooks module (so add_hookspecs sees only the spec, not the plugin's own hook impls). Returns a RerunPolicy or None (default). - RerunPolicy(reruns, all_reruns_need_to_pass, tag): fields left None keep defaults; tag is stamped on the final failed report as report.rerun_tag (survives xdist). - Called once per test on its first failing report; the protocol applies the override after the first attempt and tags the final failure. Default behavior is unchanged. - Tests, CHANGES, README, and packaging (py-modules) for the new module. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…erage Docstring fix (factual): - pytest_rerunfailures_rerun_policy report.when may be "setup", "call", OR "teardown" (the makereport guard has no when-filter); the docstring wrongly said "setup or call". Also document that the policy is locked at the first failure and not recomputed on later attempts. New tests (coverage gaps the review found): - no policy hook implemented at all -> global behavior unchanged (additive contract) - a policy can RAISE the rerun count above the global --reruns - firstresult=True: the first non-None policy wins over a later implementation - all_reruns_need_to_pass=False override takes effect with reruns>=2 (the prior tests only used reruns=1, where the flag is a no-op; mutation-tested: neutralizing the override branch makes this test fail) - policy is locked at the first failure: a differently-classed failure on a rerun inherits the first policy and is not recomputed Full suite: 152 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The tag/rerun_tag labeling was the first-generation mechanism for a consumer to identify a failure class after the run. The smith eval harness — the only consumer — never adopted it: it classifies failures per-attempt in its own conftest (report.verdict_infra, infra only if every attempt was infra) rather than reading report.rerun_tag, precisely so a behavioral regression surfacing on a rerun is never masked. Nothing else uses tag, and it was under-tested (the all-reruns tag-stamping branch + xdist serialization were unverified), so remove it. Removed: RerunPolicy.tag, _apply_rerun_tag, the rerun_tag stamping in the protocol, the tag test, and the tag docs (README/CHANGES/hookspec). The per-failure `reruns` / `all_reruns_need_to_pass` overrides — the actual infra-rerun mechanism — are unchanged and fully tested. Full suite: 151 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add pytest_rerunfailures_rerun_policy hook for per-failure rerun policy
3 participants
This addresses your requirement to verify that non-deterministic tests (that work ~90% of the time) pass consistently by requiring all reruns to pass after an initial failure.