Summary
The default built-in OpenAI judge selects gpt-5.6-luna and always sends temperature=0.0. The live OpenAI API rejects that value because this model only accepts its default temperature, so a normal judge call returns an error result before evaluation can occur.
Found while running the live end-to-end verification for #133.
Live evidence
With OPENAI_API_KEY configured and no KENSA_JUDGE_MODEL or KENSA_JUDGE_PROVIDER override, a real pytest Kensa eval called the public judge() API and received:
Error code: 400
Unsupported value: 'temperature' does not support 0.0 with this model.
Only the default (1) value is supported.
The resulting judge snapshot was:
provider=openai
model=gpt-5.6-luna
passed=false
error=true
The same live fixture passed with the Anthropic judge, which does not receive Kensa's OpenAI-only temperature override.
Cause
_LLMJudge.judge() applies temperature=0.0 to every OpenAI model:
temperature=(0.0 if self.config.provider is LLMProvider.OPENAI else None)
This is incompatible with the current default OpenAI judge model. The HTTP 400 is correctly classified as non-retryable, so the retry policy in #133 does not and should not mask it.
Impact
The zero-configuration OpenAI judge path fails every live judgment for users whose default resolves to gpt-5.6-luna. Trials receive a judge error even when the agent output is valid.
Proposed fix
Omit temperature for OpenAI models that only accept the provider default. Keep temperature handling capability-aware if Kensa continues to support models where an explicit value is useful.
Acceptance criteria
- The default OpenAI judge completes a live structured judgment without an unsupported-temperature 400
- Models that reject custom temperatures do not receive
temperature=0.0
- Genuine non-retryable HTTP 400 responses remain single-attempt failures
- Focused tests cover the default OpenAI judge arguments and structured result path
- Repository coverage remains 100%
Summary
The default built-in OpenAI judge selects
gpt-5.6-lunaand always sendstemperature=0.0. The live OpenAI API rejects that value because this model only accepts its default temperature, so a normal judge call returns an error result before evaluation can occur.Found while running the live end-to-end verification for #133.
Live evidence
With
OPENAI_API_KEYconfigured and noKENSA_JUDGE_MODELorKENSA_JUDGE_PROVIDERoverride, a real pytest Kensa eval called the publicjudge()API and received:The resulting judge snapshot was:
The same live fixture passed with the Anthropic judge, which does not receive Kensa's OpenAI-only temperature override.
Cause
_LLMJudge.judge()appliestemperature=0.0to every OpenAI model:This is incompatible with the current default OpenAI judge model. The HTTP 400 is correctly classified as non-retryable, so the retry policy in #133 does not and should not mask it.
Impact
The zero-configuration OpenAI judge path fails every live judgment for users whose default resolves to
gpt-5.6-luna. Trials receive a judge error even when the agent output is valid.Proposed fix
Omit
temperaturefor OpenAI models that only accept the provider default. Keep temperature handling capability-aware if Kensa continues to support models where an explicit value is useful.Acceptance criteria
temperature=0.0