fix(agents): downscale oversized images before they enter model context

[resumeparseeval]

Summary

A single full-resolution photo (e.g. a 4032×3024 phone image) embedded as an inline base64 data-URI is ~350K tokens — larger than the entire context budget. Because it's image data, text compaction can't shrink it, so the preemptive-overflow guard rejects the prompt on every turn:

context window exceeded: preemptive context overflow:
estimated prompt size 355796 tokens exceeds 180000 token budget (compaction disabled)

The run dies before the model is ever called, and every auto-spawned continuation inherits the same image and dies the same way — so from the UI the request just looks stuck forever. (Observed with a Facebook-Marketplace photo task on openai-codex::gpt-5.5: compaction ran and freed nothing — 355,796 → 356,014 tokens — because the bloat is an image.)

Vision models downsample internally anyway (Claude effectively caps the long edge near ~1568px), so sending full resolution buys nothing.

Fix

Cap images at the funnel where stored message content is parsed into ContentPart::Image — values_to_chat_messages_inner in crates/agents/src/model/convert.rs. That point runs before token estimation and before any provider conversion, so a single site:

• fixes the preemptive-overflow estimate (the estimator serializes the ChatMessage, so it now sees the capped image),
• covers every provider (OpenAI, Anthropic, codex, compat) without touching each conversion site,
• rescues already-oversized sessions on load.

New downscale_base64_image in multimodal.rs:

• decodes the image; if the longest edge ≤ MAX_IMAGE_EDGE (1568px) it returns None (no change),
• otherwise resizes down preserving aspect ratio (Triangle filter) and re-encodes as JPEG,
• a cheap base64-length pre-check (< 256 KB) skips the decode entirely for payloads already well under budget, so the common path pays nothing,
• swallows all failures — a botched downscale keeps the original rather than dropping the image.

A 4032×3024 photo (~350K tokens) becomes a 1568×1176 JPEG (~40–60K tokens), so the prompt fits with room for tools and history.

Adds image and base64 (both already [workspace.dependencies]) to moltis-agents.

Validation

Completed

• cargo test -p moltis-agents multimodal — 32 pass, incl. 3 new: caps an oversized 2400×1800 image to a 1568-edge JPEG and verifies it shrank; skips small payloads without decoding; leaves within-cap images untouched.
• cargo test -p moltis-agents --lib model:: — 51 pass (conversion funnel unaffected).
• cargo clippy -p moltis-agents --all-targets (pinned nightly) — clean.
• cargo fmt (pinned nightly) — changed files clean.

Remaining (CI / maintainer)

• just release-preflight, just test (full OS-aware suite).

Manual QA

1. Send a full-resolution phone photo (≥3000px) into a chat and ask the model anything about it.
2. Before: run errors with context window exceeded … exceeds 180000 token budget; the request appears stuck.
3. After: the image is downscaled to ≤1568px on load, the prompt fits, and the model responds normally. Verify in logs that estimated prompt size is now tens of thousands of tokens, not ~350K.

Notes

This is the engine-side fix for context overflow. A complementary, optional follow-up is to downscale at ingestion so the oversized base64 is never stored (reduces session/DB bloat too); this PR deliberately fixes the token/overflow problem at the load funnel since that also repairs existing sessions.

[greptile-apps]

Greptile Summary

This PR caps oversized inline images before stored messages enter model context. The main changes are:

• Added base64 and image to moltis-agents.
• Added a helper that decodes, resizes, and JPEG-encodes oversized base64 images.
• Wired stored image_url conversion to use the capped image data.
• Added direct tests for oversized, small, and already-within-cap images.

Confidence Score: 4/5

The stored-history path is improved, but the first send of a large current image can still overflow context.

• The new helper preserves image data on failures and is covered by direct tests.
• Stored image_url blocks are capped before provider conversion.
• Current inbound multimodal messages can still reach the provider before this conversion path runs.

crates/agents/src/model/convert.rs and the current-message image construction path

Important Files Changed

Filename	Overview
Cargo.lock	Records the new moltis-agents dependency edges for base64 and image.
crates/agents/Cargo.toml	Adds workspace-managed image and base64 dependencies.
crates/agents/src/model/convert.rs	Downscales stored image_url blocks during conversion, but the live current-message path can still bypass the cap.
crates/agents/src/multimodal.rs	Adds the downscale helper, constants, and direct helper tests.

_{Reviews (1): Last reviewed commit: "fix(agents): downscale oversized images ..." | Re-trigger Greptile}

[greptile-apps]

Current Images Bypass Downscale

When a user sends a new turn with a full-resolution data-URI image, the current message can be built as ContentPart::Image before this stored-history conversion runs. That first provider request can still include the original base64 payload and hit the same context overflow; the image is only capped after it is stored and later reloaded through this path.