feat: capture JudgeRubric input/output for platform Judges View

[cdreetz]

Summary

• JudgeRubric.judge() now records each call (input prompt, raw output, rubric name, model, score, timestamp) onto state["judges"]. Every call is recorded — including cache hits — so two JudgeRubric instances in a RubricGroup are both attributed.
• state_to_output auto-emits state["judges"] onto the RolloutOutput when non-empty (no state_columns opt-in needed).
• Adds an optional name= constructor arg to JudgeRubric so multiple instances can self-identify.

The companion platform PR (https://linear.app/primeintellect/issue/ENG-3552) reads the new judges field on each rollout sample and renders it in a "Judges" section in the rollout view's reward panel, with a fullscreen popup for long judge prompts.

The existing state["judge_response"] cache is preserved unchanged for backwards-compatibility.

Test plan

• tests/test_judges_view.py covers: judge call → record on state, two rubrics in a group both recorded, state_to_output propagates, absence yields no field
• uv run pytest tests/test_judges_view.py tests/test_save_utils.py tests/test_rubric.py tests/test_rubric_group.py tests/test_math_rubric.py — all green
• uv run ruff check — clean

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds a new judges payload to rollout outputs and records judge calls on every JudgeRubric.judge() invocation (including cache hits), which could affect output size/compatibility for downstream consumers expecting a stable schema.

Overview
Adds per-invocation capture of LLM-as-judge calls: JudgeRubric now appends a JudgeRecord (prompt, raw output, rubric identifier, model, timestamp) into state["judges"], with an optional name= to distinguish multiple rubric instances and recording even on cache hits.

Updates serialization so state_to_output/states_to_outputs auto-propagate non-empty state["judges"] into RolloutOutput as judges (no state_columns opt-in), and introduces the JudgeRecord type plus tests covering recording, naming, propagation, and omission when absent.

^{Reviewed by Cursor Bugbot for commit 8b58330. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8b58330. Configure here.}

[cursor]

JudgeRecord total=False makes all fields optional unexpectedly

Low Severity

JudgeRecord uses total=False which makes every field optional at the type level, but the docstring explicitly states that judge_input and judge_output are required. This mismatch means type checkers won't enforce the presence of these two critical fields, silently allowing incomplete records. Using Required[] from typing on those two fields (or splitting into a base TypedDict with total=True) would correctly express the intended contract.

 

^{Reviewed by Cursor Bugbot for commit 8b58330. Configure here.}

[cursor]

Missing documentation update for new user-facing features

Low Severity

This PR adds the name= constructor parameter to JudgeRubric and a new judges field on RolloutOutput, both user-facing. The existing docs in docs/environments.md (which shows JudgeRubric usage) and docs/reference.md (which lists RolloutOutput fields and mentions JudgeRubric) are not updated to reflect these additions, violating the documentation update rule.

Additional Locations (1)

• verifiers/types.py#L375-L376

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 8b58330. Configure here.}