Skip to content

fix: VLM image analysis for Responses-protocol providers (input_text injection) - #1849

Open
Floatend wants to merge 2 commits into
BigPizzaV3:mainfrom
Floatend:fix/vlm-responses-injection
Open

fix: VLM image analysis for Responses-protocol providers (input_text injection)#1849
Floatend wants to merge 2 commits into
BigPizzaV3:mainfrom
Floatend:fix/vlm-responses-injection

Conversation

@Floatend

Copy link
Copy Markdown

Problem

Configuring VLM image analysis (图片分析路由) for a Responses-protocol provider breaks every image request. The relay analyzes the image with the configured vision model, but then injects the description using a Chat-Completions style block:

{"type": "text", "text": "..."}

Upstream Responses APIs reject this block type, e.g. DeepSeek:

Failed to deserialize the JSON body into the target type: input: unknown variant `text`,
expected one of `input_text`, `output_text`, `input_image`, `input_file`

This is presumably why the manager UI hard-disables VLM/Strip for Responses-protocol providers (vlmUnsupportedProtocol). The rest of the VLM pipeline already handles the input array (URL collection recognizes input_image, stripping removes input_image blocks), so only the injection format is broken.

Changes

  • vision.rs: inject_text_into_message / inject_analysis / strip_image_blocks now carry a responses: bool flag and emit {"type": "input_text"} blocks for Responses requests instead of {"type": "text"}. Chat behavior unchanged.
  • protocol_proxy.rs: passes relay.protocol == RelayProtocol::Responses into strip_image_blocks.
  • App.tsx: removes the Responses-protocol disable so VLM/Strip can be configured for those providers; adds friendlier option labels (原样发送图片 / 移除图片 / 视觉辅助分析).

Tests

  • New: inject_analysis_uses_input_text_block_for_responses_protocol (input_text for responses)
  • New: inject_text_into_message_keeps_chat_text_block_when_not_responses (text preserved for chat)
  • Full suite on current main: 252 passed, 0 failed. tsc --noEmit clean.

Verification

Tested end-to-end with a DeepSeek (responses) provider + Qwen VLM: sending an image no longer errors; the image is analyzed by the VLM and the description reaches DeepSeek as text.

Note: the UI unlock (vlmUnsupportedProtocol = false) is intentional — VLM/Strip now work for Responses providers thanks to the injection fix. Chat Completions providers are unaffected.

- inject_text_into_message now emits input_text blocks for Responses API
  (DeepSeek rejects chat-style 'text' blocks with deserialization error)
- strip_image_blocks / inject_analysis carry a responses flag through
- unlock VLM analysis in the manager UI for Responses-protocol providers
- add tests: input_text injection (responses) vs text (chat)
@BigPizzaV3

Copy link
Copy Markdown
Owner

维护者审查:修复方向正确。Responses 请求中的文本内容块应使用 input_text,当前改动也保留了 Chat Completions 的 text 行为。已批准来自 fork 的三平台 CI。

合并前建议再补一项完整链路回归测试:构造含 input_image 的 Responses 请求,经 strip_image_blocks 处理后断言图片块被移除、VLM 描述以 input_text 注入,并且最终请求体不再包含 Chat 格式的新增 text 块。当前新增测试只直接调用 inject_analysis,尚未覆盖 protocol_proxy -> strip_image_blocks 的协议标志传递。CI 通过且该链路测试补齐后可以合并。

…test

Covers the full strip_image_blocks path with input_image blocks:
- input_image stripped
- VLM description injected as input_text (not chat-style text)
- no 'text' blocks introduced for Responses requests
@Floatend

Copy link
Copy Markdown
Author

已补齐完整链路回归测试,见新提交 4f291cd:

strip_image_blocks_injects_input_text_for_responses_protocol(wiremock 真实 HTTP):

  1. 构造含 input_image 块的 Responses 格式请求(与 Codex 发往中继的格式一致)
  2. strip_image_blocks(..., responses=true) 完整链路处理(mock VLM 返回描述)
  3. 断言:图片块被移除 / 描述以 input_text 注入 / 请求体不出现 Chat 格式的 text

该测试覆盖了 protocol_proxy → strip_image_blocks 的协议标志传递对应的行为。测试结果:

  • lib 测试 217 passed, 0 failed(新增 1 个)
  • 全量 cargo test:lib 217 + integration 36,全部通过(唯一失败项为 Windows 环境 symlink 权限测试,与本次改动无关)
  • tsc --noEmit 通过

CI 结果出来后可随时合并,多谢 review!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants