fix: VLM image analysis for Responses-protocol providers (input_text injection) - #1849
Open
Floatend wants to merge 2 commits into
Open
fix: VLM image analysis for Responses-protocol providers (input_text injection)#1849Floatend wants to merge 2 commits into
Floatend wants to merge 2 commits into
Conversation
- inject_text_into_message now emits input_text blocks for Responses API (DeepSeek rejects chat-style 'text' blocks with deserialization error) - strip_image_blocks / inject_analysis carry a responses flag through - unlock VLM analysis in the manager UI for Responses-protocol providers - add tests: input_text injection (responses) vs text (chat)
Owner
|
维护者审查:修复方向正确。Responses 请求中的文本内容块应使用 合并前建议再补一项完整链路回归测试:构造含 |
…test Covers the full strip_image_blocks path with input_image blocks: - input_image stripped - VLM description injected as input_text (not chat-style text) - no 'text' blocks introduced for Responses requests
Author
|
已补齐完整链路回归测试,见新提交
该测试覆盖了 protocol_proxy → strip_image_blocks 的协议标志传递对应的行为。测试结果:
CI 结果出来后可随时合并,多谢 review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Configuring VLM image analysis (图片分析路由) for a Responses-protocol provider breaks every image request. The relay analyzes the image with the configured vision model, but then injects the description using a Chat-Completions style block:
{"type": "text", "text": "..."}Upstream Responses APIs reject this block type, e.g. DeepSeek:
This is presumably why the manager UI hard-disables VLM/Strip for Responses-protocol providers (
vlmUnsupportedProtocol). The rest of the VLM pipeline already handles theinputarray (URL collection recognizesinput_image, stripping removesinput_imageblocks), so only the injection format is broken.Changes
vision.rs:inject_text_into_message/inject_analysis/strip_image_blocksnow carry aresponses: boolflag and emit{"type": "input_text"}blocks for Responses requests instead of{"type": "text"}. Chat behavior unchanged.protocol_proxy.rs: passesrelay.protocol == RelayProtocol::Responsesintostrip_image_blocks.App.tsx: removes the Responses-protocol disable so VLM/Strip can be configured for those providers; adds friendlier option labels (原样发送图片 / 移除图片 / 视觉辅助分析).Tests
inject_analysis_uses_input_text_block_for_responses_protocol(input_text for responses)inject_text_into_message_keeps_chat_text_block_when_not_responses(text preserved for chat)tsc --noEmitclean.Verification
Tested end-to-end with a DeepSeek (responses) provider + Qwen VLM: sending an image no longer errors; the image is analyzed by the VLM and the description reaches DeepSeek as text.
Note: the UI unlock (
vlmUnsupportedProtocol = false) is intentional — VLM/Strip now work for Responses providers thanks to the injection fix. Chat Completions providers are unaffected.