Skip to content

fix(judge): omit unsupported temperature for OpenAI reasoning models #135

Description

@satyaborg

Summary

The default built-in OpenAI judge selects gpt-5.6-luna and always sends temperature=0.0. The live OpenAI API rejects that value because this model only accepts its default temperature, so a normal judge call returns an error result before evaluation can occur.

Found while running the live end-to-end verification for #133.

Live evidence

With OPENAI_API_KEY configured and no KENSA_JUDGE_MODEL or KENSA_JUDGE_PROVIDER override, a real pytest Kensa eval called the public judge() API and received:

Error code: 400
Unsupported value: 'temperature' does not support 0.0 with this model.
Only the default (1) value is supported.

The resulting judge snapshot was:

provider=openai
model=gpt-5.6-luna
passed=false
error=true

The same live fixture passed with the Anthropic judge, which does not receive Kensa's OpenAI-only temperature override.

Cause

_LLMJudge.judge() applies temperature=0.0 to every OpenAI model:

temperature=(0.0 if self.config.provider is LLMProvider.OPENAI else None)

This is incompatible with the current default OpenAI judge model. The HTTP 400 is correctly classified as non-retryable, so the retry policy in #133 does not and should not mask it.

Impact

The zero-configuration OpenAI judge path fails every live judgment for users whose default resolves to gpt-5.6-luna. Trials receive a judge error even when the agent output is valid.

Proposed fix

Omit temperature for OpenAI models that only accept the provider default. Keep temperature handling capability-aware if Kensa continues to support models where an explicit value is useful.

Acceptance criteria

  • The default OpenAI judge completes a live structured judgment without an unsupported-temperature 400
  • Models that reject custom temperatures do not receive temperature=0.0
  • Genuine non-retryable HTTP 400 responses remain single-attempt failures
  • Focused tests cover the default OpenAI judge arguments and structured result path
  • Repository coverage remains 100%

Metadata

Metadata

Assignees

No one assigned

    Labels

    fixBug fixes

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions