Skip to content

docs: Explain how reviewer tool evidence affects resumed reviews #551

Description

@zzwong

Small documentation follow-up to #540, where the runtime fix landed and the artifact documentation was explicitly deferred.

The useful thing to explain is why a reviewer task can succeed while its review still cannot approve. For example, this fragment of metadata.json is possible:

{
  "status": "succeeded",
  "reviewer_tool_evidence": {
    "diff_status": "failed",
    "diff_diagnostic": "fixed diff unavailable"
  }
}

Here, succeeded means the task produced valid structured output. The failed diff tool still makes reviewer coverage incomplete_tool and clamps an otherwise approving result to comment; the existing regression test covers this. The artifact guide currently omits reviewer_tool_evidence, so this distinction requires reading the code.

A focused update to docs/llm-task-artifacts.md should be enough:

  • Describe diff_status (not_invoked, incomplete, succeeded, failed) and optional diff_diagnostic, following the existing type definitions.
  • Explain its effect on coverage and approval, and why cached tasks restore it from metadata. That gives the guide a concrete explanation of why the field is behaviorally significant.
  • Describe the current version-1 compatibility behavior: an absent field means no adapter-provided evidence; it is distinct from an explicit not_invoked result. Without evidence, the normal coverage checks still apply.

Done when the guide explains the example above, preservation on resume, and the missing-field case. A short schema entry and example can close this documentation follow-up.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions