Skip to content

fix: guard against None content in JudgeRubric judge response#1312

Open
vominh1919 wants to merge 1 commit intoPrimeIntellect-ai:mainfrom
vominh1919:fix/judge-rubric-none-content-deref
Open

fix: guard against None content in JudgeRubric judge response#1312
vominh1919 wants to merge 1 commit intoPrimeIntellect-ai:mainfrom
vominh1919:fix/judge-rubric-none-content-deref

Conversation

@vominh1919
Copy link
Copy Markdown

@vominh1919 vominh1919 commented May 8, 2026

Problem

In verifiers/rubrics/judge_rubric.py:98, the judge response is extracted with:

judge_response = str(judge_response.choices[0].message.content)

When the judge model returns tool_calls or a refusal (e.g., content filter), message.content is None. In Python, str(None) produces the literal string "None", which is then parsed as the judge's scoring response. This causes:

  1. Silent incorrect scoring — the judge "evaluates" the string "None" instead of the actual response
  2. No error signal — the caller has no idea the judge failed; it just gets a wrong score
  3. Hard-to-debug training issues — if using this for RL training, the wrong rewards silently corrupt the training signal

Fix

Check for None content and raise a clear RuntimeError with diagnostic info:

content = judge_response.choices[0].message.content
if content is None:
    raise RuntimeError(
        f"Judge model returned None content. "
        f"This usually means the model returned tool_calls or a refusal. "
        f"Model: {self.judge_model}"
    )
judge_response = str(content)

The error is raised inside the existing try/except Exception block (line 126), so it will be caught and logged properly with the model name.

Before vs After

Scenario Before After
Normal response Works correctly Works correctly
content = None str(None) = "None" → silent wrong score RuntimeError → logged + propagated
Tool calls only str(None) = "None" → silent wrong score RuntimeError → logged + propagated

Tests

This change only affects the error path when content is None. The happy path (non-None content) is completely unchanged. Existing tests that mock the judge client with valid responses will continue to pass.


Note

Medium Risk
Changes judge scoring behavior to raise an exception when the model returns None content (e.g., tool calls/refusals), which may surface new runtime failures for callers that previously received a silent "None" string.

Overview
Prevents silent mis-scoring in JudgeRubric by validating the judge model response before parsing.

When choices[0].message.content is None (commonly due to tool calls or refusals), the code now raises a descriptive RuntimeError instead of converting None to the literal string "None" and caching/returning it.

Reviewed by Cursor Bugbot for commit 2e8644b. Bugbot is set up for automated code reviews on this repo. Configure here.

When the judge model returns tool_calls or a refusal, message.content
is None. The previous code did str(None) which produced the string
"None", causing the judge to silently score the literal string "None"
instead of raising an error.

Now raises RuntimeError with a clear message when content is None,
so the caller can handle the failure appropriately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant