test: record real gateway tool loop cassettes (OpenAI + vLLM)#94
Open
ashwing wants to merge 1 commit into
Open
test: record real gateway tool loop cassettes (OpenAI + vLLM)#94ashwing wants to merge 1 commit into
ashwing wants to merge 1 commit into
Conversation
Replace the hand-authored composed cassette (merged with vllm-project#83) with real recorded traffic driven through the streaming loop, per review feedback on vllm-project#83 ("keep cassettes true source of data not stub"). Recorded via the repo's record_cassette.py against OpenAI (gpt-4o) and local vLLM (openai/gpt-oss-20b), mirroring the dual-backend approach in vllm-project#77. Cassettes + tests, each replaying real typed SSE: - web_search -> codex namespace across turns (OpenAI + vLLM): gateway Continue then client RequiresClientAction with the flat agentic_ns__ name restored to {namespace, name}. - multi-round gateway (OpenAI + vLLM): web_search on two consecutive rounds then a final message -- the only coverage of the loop's Continue decision firing more than once. - same-turn mixed (OpenAI only): web_search + namespace in one model output -> gateway executes and the turn hands back in a single round (precedence). gpt-oss does not emit both tools in one turn, so this path is OpenAI-only. Recorder masks auth headers; no credentials in the cassettes. Signed-off-by: Ashwin Giridharan <girida@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follows up on the review thread on #83 — replaces the hand-authored composed cassette (which merged with #83) with real recorded traffic, per @maralbahari's ask to keep cassettes a true source of data rather than stub values.
Recorded via the repo's own
record_cassette.pyagainst two backends — OpenAI (gpt-4o) and local vLLM (openai/gpt-oss-20b) — mirroring the dual-backend approach in #77. Every turn is real typed SSE driven through the streaming loop; the recorder masks auth headers, so no credentials are in the cassettes.Three distinct loop paths, each a
dispatch_loop_cassette_testcase:Continue→ clientRequiresClientAction; flatagentic_ns__…restored to{namespace, name}web_searchon two consecutive rounds then a final message — the only coverage of the loop'sContinuedecision firing more than onceweb_search+ namespace in one model output → gateway executes and the turn hands back in a single round (client-owned precedence)The same-turn path is OpenAI-only by necessity:
gpt-oss-20bdoes not emit both tools in one turn, so a real vLLM recording of it isn't capturable (documented on the test).Test Plan
cargo test --workspace— 334 pass, 0 fail (8 indispatch_loop_cassette_test)cargo clippy --all-targets -- -D warnings— cleancargo fmt --check— clean