Skip to content

test: record real gateway tool loop cassettes (OpenAI + vLLM)#94

Open
ashwing wants to merge 1 commit into
vllm-project:mainfrom
ashwing:test/codex-web-search-cassettes
Open

test: record real gateway tool loop cassettes (OpenAI + vLLM)#94
ashwing wants to merge 1 commit into
vllm-project:mainfrom
ashwing:test/codex-web-search-cassettes

Conversation

@ashwing

@ashwing ashwing commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Follows up on the review thread on #83 — replaces the hand-authored composed cassette (which merged with #83) with real recorded traffic, per @maralbahari's ask to keep cassettes a true source of data rather than stub values.

Recorded via the repo's own record_cassette.py against two backends — OpenAI (gpt-4o) and local vLLM (openai/gpt-oss-20b) — mirroring the dual-backend approach in #77. Every turn is real typed SSE driven through the streaming loop; the recorder masks auth headers, so no credentials are in the cassettes.

Three distinct loop paths, each a dispatch_loop_cassette_test case:

Path Backends What it exercises
web_search → codex namespace across turns OpenAI + vLLM gateway Continue → client RequiresClientAction; flat agentic_ns__… restored to {namespace, name}
multi-round gateway OpenAI + vLLM web_search on two consecutive rounds then a final message — the only coverage of the loop's Continue decision firing more than once
same-turn mixed OpenAI only web_search + namespace in one model output → gateway executes and the turn hands back in a single round (client-owned precedence)

The same-turn path is OpenAI-only by necessity: gpt-oss-20b does not emit both tools in one turn, so a real vLLM recording of it isn't capturable (documented on the test).

Test Plan

  • cargo test --workspace — 334 pass, 0 fail (8 in dispatch_loop_cassette_test)
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo fmt --check — clean
  • Cassettes recorded against live OpenAI + vLLM; verified no API keys present in the recorded YAML

Replace the hand-authored composed cassette (merged with vllm-project#83) with real
recorded traffic driven through the streaming loop, per review feedback on
vllm-project#83 ("keep cassettes true source of data not stub"). Recorded via the repo's
record_cassette.py against OpenAI (gpt-4o) and local vLLM (openai/gpt-oss-20b),
mirroring the dual-backend approach in vllm-project#77.

Cassettes + tests, each replaying real typed SSE:
- web_search -> codex namespace across turns (OpenAI + vLLM): gateway Continue
  then client RequiresClientAction with the flat agentic_ns__ name restored to
  {namespace, name}.
- multi-round gateway (OpenAI + vLLM): web_search on two consecutive rounds then
  a final message -- the only coverage of the loop's Continue decision firing
  more than once.
- same-turn mixed (OpenAI only): web_search + namespace in one model output ->
  gateway executes and the turn hands back in a single round (precedence).
  gpt-oss does not emit both tools in one turn, so this path is OpenAI-only.

Recorder masks auth headers; no credentials in the cassettes.

Signed-off-by: Ashwin Giridharan <girida@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant