feat(server): use shared model-agnostic canonical request/response types#307
Open
sercand wants to merge 4 commits into
Open
feat(server): use shared model-agnostic canonical request/response types#307sercand wants to merge 4 commits into
sercand wants to merge 4 commits into
Conversation
Introduce crates/spark-server/src/ir/ — a provider-free chat IR (ChatRequest, Message with content always a list of parts, ChatResponse, StreamDelta) — and migrate every API surface onto it: openai/to_ir.rs + anthropic/to_ir.rs adapt requests in; encode.rs / encode_stream.rs / translate.rs / responses_translate.rs serialize responses out. Deletes the Anthropic struct→JSON→struct round-trip, the Responses-API image drop, and the <think> / [tool error] string prefixes (now real fields). Images ride every role uniformly, including tool results (issue Avarok-Cybersecurity#165). Also scopes the post-think EOS guard to force-closed </think> spans (sticky ActiveSeq::think_force_closed): it previously fired on every tools-armed turn, eating the model's natural <|im_end|> after brief post-tool answers and causing scaffold garbage + spurious duplicate tool calls. Verified live on Qwen3.6-35B-A3B-NVFP4: turn-2 /v1/messages replays 4/4 clean end_turn; 2-turn tool+image agent loop 3/3 clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
All contributors have signed the CLA. Thank you! |
Author
|
I have read the CLA Document and I hereby sign the CLA |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is quite big, sorry for putting everything into single PR.
Summary
Introduces a canonical, model-agnostic chat IR (
crates/spark-server/src/ir/) and migrates every API surface to it, so the core pipeline (build_msg_entries, template rendering, scheduler) consumes one provider-free representation instead of the OpenAI wire structs. Each protocol becomes a thin, unit-tested adapter pair:openai/to_ir.rs/anthropic/to_ir.rson the request side,openai/encode.rs+encode_stream.rs,anthropic/translate.rs(ir_to_anthropic_response) + thetranslator.rsstreaming state machine, and typedresponses_translate.rs/responses_stream.rson the response side.Because
Message.contentis now always a list of parts for every role, the motivating bug — images attached to tool-result messages being silently dropped — falls out for free, along with the rest of the OpenAI-shape tax: the Anthropic struct→JSON→struct round-trip is deleted, the Responses API no longer drops images, and the<think>\n\n</think>and[tool error]\nstring-smuggling hacks become real fields (reasoning, an error flag on tool results). The migration commit also fixes a scheduler bug surfaced by end-to-end verification of the tool+image agent loop: the post-think EOS guard fired on every tools-armed turn (eating the model's natural<|im_end|>and causing scaffold garbage plus spurious duplicate tool calls); it is now gated on a stickythink_force_closedflag so it only applies after a watchdog/budget force-injected</think>.Closes #165 and #164
Test plan
cargo fmt --all -- --checkATLAS_SKIP_BUILD=1 cargo clippy --workspace --tests -- -Dwarningsbash scripts/check-license-headers.sh/v1/messagesturn-2 tool replays 4/4 cleanend_turn; full 2-turn tool+image agent loop 3/3 clean (previously 4 turns with chat-template scaffold garbage and duplicate tool calls); newscripts/test-qwen36-tool-image.shdrives the loop end-to-endcargo test -p spark-server→ 689 passed, 0 failed (30 lib + 659 bin). New suites:ir/tests.rs,anthropic/tests/ir_carry.rs(wire→IR→wire carry),anthropic/tests/translator_stream.rs(streaming-framing goldens), expandedopenai/tests.rs; prompt-byte stability gates (prompt_json_stability_gate,rendered_prompt_json_matches_retired_json_hop_output) pin rendered prompts to the pre-migration outputNotes for reviewers
service_tier,metadata,store,include_usage) deliberately do not enter the IR; they rideapi/chat/echo.rs::ResponseEchohandler-side.openai/before hitting the IR — intra-provider normalization, deliberate; it now carries images instead of dropping them (from_responses_input_item).response_store(caches wire bodies) andchat_stream --dumpsynthesis.ApiErrortype; top-levelreasoning_effortremains dead wire-compat (never wired into the thinking ladder).count_tokensnow sharesapi/chat/prepare.rs::prepare_chat_promptwith the serving path (measured end-to-end drift 0.0%).CLA