Skip to content

fix(media): downsample oversized image model inputs - #3965

Merged
chenjw merged 2 commits into
volcengine:mainfrom
MaojiaSheng:main
Aug 12, 2026
Merged

fix(media): downsample oversized image model inputs#3965
chenjw merged 2 commits into
volcengine:mainfrom
MaojiaSheng:main

Conversation

@MaojiaSheng

Copy link
Copy Markdown
Collaborator

Description

Keep imported image resources unchanged while avoiding provider-side multimodal embedding failures for oversized images. The embedding path now builds a temporary downsampled image data URI when image bytes exceed the shared large-image limits.

Move reusable image size thresholds into media_limits so both parser-side large image handling and embedding-side input preparation depend on a common utility instead of embedding_utils importing parser internals.

Add vectorize_file coverage confirming large image embedding inputs are resized and the stored resource bytes are preserved.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Performance improvement
  • Test update

Changes Made

  • Add shared image model-input preparation in openviking/utils/image_search.py
    • Detects oversized images using the existing large-image limits
    • Downsamples image bytes in memory for model calls
    • Leaves stored resource bytes unchanged
  • Apply the shared downsampling path to semantic image summary generation
    • Prevents generate_image_summary() from sending oversized raw image bytes to VLM providers
  • Apply the same shared downsampling path to image embedding inputs
    • Keeps embedding_utils.py from owning parser/media-specific resizing logic
  • Add test coverage for VLM image summaries
    • Verifies model input is resized
    • Verifies original stored bytes are preserved

Testing

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested this on the following platforms:
    • Linux
    • macOS
    • Windows

Checklist

  • My code follows the project's coding style
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Screenshots (if applicable)

image

Additional Notes

Keep imported image resources unchanged while avoiding provider-side multimodal embedding failures for oversized images. The embedding path now builds a temporary downsampled image data URI when image bytes exceed the shared large-image limits.

Move reusable image size thresholds into media_limits so both parser-side large image handling and embedding-side input preparation depend on a common utility instead of embedding_utils importing parser internals.

Add vectorize_file coverage confirming large image embedding inputs are resized and the stored resource bytes are preserved.

Co-authored-by: TRAE CLI <noreply@bytedance.com>
Comment thread openviking/utils/embedding_utils.py Outdated
Comment thread openviking/utils/embedding_utils.py Outdated
Comment thread openviking/utils/media_limits.py Outdated
Keep imported image resources unchanged while avoiding provider-side multimodal failures for oversized images. Shared image input preparation now builds temporary downsampled bytes for model requests when image bytes exceed the configured large-image limits.

Apply the model-input downsampling to both semantic image summary generation and embedding image data URI construction, so directory and code repository imports can preserve original images while sending provider-compatible inputs.

Move reusable image size thresholds into media_limits so parser-side large image handling, VLM summary generation, and embedding preparation share common limits without embedding_utils importing parser internals.

Always convert downsampled model images to RGB before JPEG encoding so Pillow-openable modes such as LA or I;16 do not fall back to the original oversized bytes.

Add coverage confirming VLM image summaries, vectorize_file embedding inputs, and JPEG-incompatible image modes are resized while stored resource bytes are preserved.

Co-authored-by: TRAE CLI <noreply@bytedance.com>
@chenjw
chenjw merged commit 5aed7f7 into volcengine:main Aug 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

2 participants